[PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs opened a new pull request, #45360:
URL: https://github.com/apache/arrow/pull/45360

   <!--
   Thanks for opening a pull request!
   If this is your first pull request you can find detailed information on how 
   to contribute here:
     * [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
     * [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
   
   
   If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
   
   Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
   
   Then could you also rename the pull request title in the following format?
   
       GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
   
   or
   
       MINOR: [${COMPONENT}] ${SUMMARY}
   
   -->
   
   ### Rationale for this change
   
   I have been working on to improve Parquet's deduplication efficiency for content-addressable storages. These system generally use some kind of a [CDC algorithm](https://joshleeb.com/posts/content-defined-chunking.html) which are better suited for uncompressed row-major formats. Although thanks to Parquet's unique features I was able to reach good deduplication results by consistently chunking data pages by maintaining a gearhash based chunker for each column. 
   
   A purpose built evaluation tool is available at https://github.com/kszucs/de
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   ### What changes are included in this PR?
   
   
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   ### Are these changes tested?
   
   Not yet. 
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   ### Are there any user-facing changes?
   
   There is a new parquet writer property called `content_defined_chunking` which is subject to renaming.
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
   -->
   <!-- **This PR includes breaking changes to public APIs.** -->
   
   <!--
   Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
   -->
   <!-- **This PR contains a "Critical Fix".** -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kou (via GitHub)" <gi...@apache.org>]

kou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1973016269


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }

Review Comment:
   I'm not sure whether this is a critical path or not but we may be able to eliminate this branch by `has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) ==0)`. This may improve performance.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kou (via GitHub)" <gi...@apache.org>]

kou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1974573027


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {

Review Comment:
   Yes. I think that "in the same directory" is the right place.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kou (via GitHub)" <gi...@apache.org>]

kou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1974576805


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,773 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {

Review Comment:
   ```suggestion
   constexpr uint64_t GEARHASH_TABLE[8][256] = {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2014039039


##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1332,13 +1337,38 @@ class TypedColumnWriterImpl : public ColumnWriterImpl, public TypedColumnWriter<
       bits_buffer_->ZeroPadding();
     }
 
-    if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
-      return WriteArrowDictionary(def_levels, rep_levels, num_levels, leaf_array, ctx,
-                                  maybe_parent_nulls);
+    if (this->properties_->cdc_enabled()) {
+      ARROW_ASSIGN_OR_RAISE(auto boundaries,
+                            content_defined_chunker_.GetBoundaries(
+                                def_levels, rep_levels, num_levels, leaf_array));
+      for (auto chunk : boundaries) {
+        auto chunk_array = leaf_array.Slice(chunk.value_offset);

Review Comment:
   Good idea, I added a `ContentDefinedChunker::Impl::ValidateChunks()` method called from the chunking implementation to do sanity checks in debug builds.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["mapleFU (via GitHub)" <gi...@apache.org>]

mapleFU commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2013720625


##########
cpp/src/parquet/properties.h:
##########
@@ -260,7 +268,9 @@ class PARQUET_EXPORT WriterProperties {
           created_by_(DEFAULT_CREATED_BY),
           store_decimal_as_integer_(false),
           page_checksum_enabled_(false),
-          size_statistics_level_(DEFAULT_SIZE_STATISTICS_LEVEL) {}
+          size_statistics_level_(DEFAULT_SIZE_STATISTICS_LEVEL),
+          content_defined_chunking_enabled_(false),

Review Comment:
   Current impl also LGTM



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2766424739

   > I'm a little busy today and will take a careful round tomorrow
   
   Thank you! Hopefully the `20.0.0` feature freeze date `April 1` is inclusive :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1981306642


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+bool ContentDefinedChunker::Check() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last Check() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling
+    // hash state here, so the next Check() call will continue from the current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename T>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const T& leaf_array) {
+  std::vector<Chunk> result;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(def_levels[offset]);
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else {
+    // nested data with nulls
+    bool has_leaf_value;
+    bool is_record_boundary;
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t level_offset = 0;
+    int64_t value_offset = 0;
+    int64_t record_level_offset = 0;
+    int64_t record_value_offset = 0;
+
+    while (level_offset < num_levels) {
+      def_level = def_levels[level_offset];
+      rep_level = rep_levels[level_offset];
+
+      has_leaf_value = def_level >= level_info_.repeated_ancestor_def_level;
+      is_record_boundary = rep_level == 0;
+
+      Roll(def_level);
+      Roll(rep_level);
+      if (has_leaf_value) {
+        Roll(leaf_array.GetView(value_offset));
+      }
+
+      if (is_record_boundary && Check()) {
+        auto levels_to_write = level_offset - record_level_offset;
+        if (levels_to_write > 0) {
+          result.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+          record_level_offset = level_offset;
+          record_value_offset = value_offset;
+        }
+      }
+
+      ++level_offset;
+      if (has_leaf_value) {
+        ++value_offset;
+      }
+    }
+
+    auto levels_to_write = num_levels - record_level_offset;
+    if (levels_to_write > 0) {
+      result.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+    }
+  }
+
+  return result;
+}
+
+#define PRIMITIVE_CASE(TYPE_ID, ArrowType)               \
+  case ::arrow::Type::TYPE_ID:                           \
+    return Calculate(def_levels, rep_levels, num_levels, \
+                     static_cast<const ::arrow::ArrowType##Array&>(values));
+
+const ::arrow::Result<std::vector<Chunk>> ContentDefinedChunker::GetBoundaries(
+    const int16_t* def_levels, const int16_t* rep_levels, int64_t num_levels,
+    const ::arrow::Array& values) {
+  auto type_id = values.type()->id();
+  switch (type_id) {

Review Comment:
   `VisitArrayInline()` would be certainly better, but I would need to disable the implementations for unsupported types, like nested ones. Also passing the result as argument + arrow status handling would add up more noise. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kou (via GitHub)" <gi...@apache.org>]

kou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1976514402


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+bool ContentDefinedChunker::Check() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last Check() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling
+    // hash state here, so the next Check() call will continue from the current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename T>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const T& leaf_array) {
+  std::vector<Chunk> result;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(def_levels[offset]);
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else {
+    // nested data with nulls
+    bool has_leaf_value;
+    bool is_record_boundary;
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t level_offset = 0;
+    int64_t value_offset = 0;
+    int64_t record_level_offset = 0;
+    int64_t record_value_offset = 0;
+
+    while (level_offset < num_levels) {
+      def_level = def_levels[level_offset];
+      rep_level = rep_levels[level_offset];
+
+      has_leaf_value = def_level >= level_info_.repeated_ancestor_def_level;
+      is_record_boundary = rep_level == 0;
+
+      Roll(def_level);
+      Roll(rep_level);
+      if (has_leaf_value) {
+        Roll(leaf_array.GetView(value_offset));
+      }
+
+      if (is_record_boundary && Check()) {
+        auto levels_to_write = level_offset - record_level_offset;
+        if (levels_to_write > 0) {
+          result.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+          record_level_offset = level_offset;
+          record_value_offset = value_offset;
+        }
+      }
+
+      ++level_offset;
+      if (has_leaf_value) {
+        ++value_offset;
+      }
+    }
+
+    auto levels_to_write = num_levels - record_level_offset;
+    if (levels_to_write > 0) {
+      result.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+    }
+  }
+
+  return result;
+}
+
+#define PRIMITIVE_CASE(TYPE_ID, ArrowType)               \
+  case ::arrow::Type::TYPE_ID:                           \
+    return Calculate(def_levels, rep_levels, num_levels, \
+                     static_cast<const ::arrow::ArrowType##Array&>(values));
+
+const ::arrow::Result<std::vector<Chunk>> ContentDefinedChunker::GetBoundaries(
+    const int16_t* def_levels, const int16_t* rep_levels, int64_t num_levels,
+    const ::arrow::Array& values) {
+  auto type_id = values.type()->id();
+  switch (type_id) {

Review Comment:
   Oh, sorry.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1985159947


##########
cpp/src/parquet/chunker_internal_hashtable.py:
##########
@@ -0,0 +1,90 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import hashlib
+import pathlib
+import sys
+from io import StringIO
+
+
+template = """\
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+#include <cstdint>
+
+namespace parquet {{
+namespace internal {{
+
+constexpr uint64_t GEARHASH_TABLE[8][256] = {{
+{content}}};
+
+}}  // namespace internal
+}}  // namespace parquet
+"""
+
+
+def generate_hash(n: int, seed: int):

Review Comment:
   Added some docs and also references to two other implementations. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1983655666


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {

Review Comment:
   Hmm, shouldn't you fix the test assertions then? They shouldn't depend on the _exact_ random-generated data.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1995763715


##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage system (CAS)
+/// which splits the bytes stream into content defined blobs. The CAS system will
+/// calculate a unique identifier for each blob, then store the blob in a key-value store.
+/// If the same blob is encountered again, the system can refer to the hash instead of
+/// physically storing the blob again. In the example above, the CAS system would store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks.
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data
+///   Deduplication
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+/// - Git is for Data (chunk size normalization used here is described in section 6.2.1):
+///   https://www.cidrdb.org/cidr2023/papers/p43-low.pdf
+class ContentDefinedChunker {
+ public:
+  /// Create a new ContentDefinedChunker instance
+  ///
+  /// @param level_info Information about definition and repetition levels
+  /// @param size_range Min/max chunk size as pair<min_size, max_size>, the chunker will
+  ///                   attempt to uniformly distribute the chunks between these extremes.
+  /// @param norm_factor Normalization factor to center the chunk size around the average
+  ///                    size more aggressively. By increasing the normalization factor,
+  ///                    probability of finding a chunk boundary increases.
+  ContentDefinedChunker(const LevelInfo& level_info,
+                        std::pair<uint64_t, uint64_t> size_range,
+                        uint8_t norm_factor = 0);
+
+  /// Get the chunk boundaries for the given column data
+  ///
+  /// @param def_levels Definition levels
+  /// @param rep_levels Repetition levels
+  /// @param num_levels Number of levels
+  /// @param values Column values as an Arrow array
+  /// @return Vector of Chunk objects representing the chunk boundaries
+  const std::vector<Chunk> GetBoundaries(const int16_t* def_levels,
+                                         const int16_t* rep_levels, int64_t num_levels,
+                                         const ::arrow::Array& values);
+
+ private:
+  // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+  // true if the hash matches the mask.
+  template <typename T>
+  void Roll(const T value);
+

Review Comment:
   Updated, note that I'm planning a follow-up to unify the fixed-sized-chunking logic used at `DoInBatches` and the CDC chunker implemented here behind a common chunker API. Data page adding happens on different conditions, so it is not entirely trivial to refactor but I believe it is doable. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1995777717


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             int64_t min_size, int64_t max_size,
+                                             int8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(min_size),
+      max_size_(max_size),
+      hash_mask_(GetMask(min_size, max_size, norm_factor)) {
+  if (min_size_ < 0) {
+    throw ParquetException("min_size must be non-negative");
+  }
+  if (max_size_ < 0) {
+    throw ParquetException("max_size must be non-negative");
+  }
+  if (min_size_ > max_size_) {
+    throw ParquetException("min_size must be less than or equal to max_size");
+  }
+}
+
+void ContentDefinedChunker::Roll(const bool value) {
+  if (chunk_size_++ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+  has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+}
+
+template <int ByteWidth>
+void ContentDefinedChunker::Roll(const uint8_t* value) {
+  chunk_size_ += ByteWidth;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (size_t i = 0; i < ByteWidth; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T* value) {
+  return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+}
+
+void ContentDefinedChunker::Roll(const uint8_t* value, int64_t length) {
+  chunk_size_ += length;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (auto i = 0; i < length; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, note that we don't reset the
+    // rolling hash state here, so the next NeedNewChunk() call will continue from the
+    // current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename RollFunc>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const RollFunc& RollValue) {
+  std::vector<Chunk> chunks;
+  int64_t offset;
+  int64_t prev_offset = 0;
+  int64_t prev_value_offset = 0;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    for (offset = 0; offset < num_levels; ++offset) {
+      RollValue(offset);
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int16_t def_level;
+    for (int64_t offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+
+      Roll(&def_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(offset);
+      }
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else {
+    // nested data with nulls
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t value_offset = 0;
+
+    for (offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+      rep_level = rep_levels[offset];
+
+      Roll(&def_level);
+      Roll(&rep_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(value_offset);
+      }
+
+      if ((rep_level == 0) && NeedNewChunk()) {
+        // if we are at a record boundary and need a new chunk, we create a new chunk
+        auto levels_to_write = offset - prev_offset;
+        if (levels_to_write > 0) {
+          chunks.emplace_back(prev_offset, prev_value_offset, levels_to_write);
+          prev_offset = offset;
+          prev_value_offset = value_offset;
+        }
+      }
+      if (def_level >= level_info_.repeated_ancestor_def_level) {
+        // we only increment the value offset if we have a leaf value
+        ++value_offset;
+      }
+    }
+  }
+
+  // add the last chunk if we have any levels left
+  if (prev_offset < num_levels) {
+    chunks.emplace_back(prev_offset, prev_value_offset, num_levels - prev_offset);
+  }
+  return chunks;
+}
+
+#define FIXED_WIDTH_CASE(ByteWidth)                                       \
+  {                                                                       \
+    const auto raw_values = values.data()->GetValues<uint8_t>(1);         \

Review Comment:
   Ahh, it depends on the requested `T` and we are requesting `uint8_t`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086963427


##########
cpp/src/parquet/properties.h:
##########
@@ -245,6 +245,34 @@ class PARQUET_EXPORT ColumnProperties {
   bool page_index_enabled_;
 };
 
+// EXPERIMENTAL: Options for content-defined chunking.
+struct PARQUET_EXPORT CdcOptions {
+  /// Minimum chunk size in bytes, default is 256 KiB

Review Comment:
   Added more doctoring to clarify your question.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2876833860

   Revision: 1cc2e4b690e467f548dc4131f9b392a864a132aa
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-6a43d39b56](https://github.com/ursacomputing/crossbow/branches/all?query=actions-6a43d39b56)
   
   |Task|Status|
   |----|------|
   |test-conda-cpp-valgrind|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6a43d39b56-github-test-conda-cpp-valgrind)](https://github.com/ursacomputing/crossbow/actions/runs/14999702661/job/42143066837)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086224113


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,412 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+namespace {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_level parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_level Normalization level (default 0)
+// @return The mask used to compare against the rolling hash
+uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size, int norm_level) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be non-negative");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes; also divide by the number of gearhash tables to have a
+  // a more gaussian-like distribution
+  int64_t target_size = (avg_chunk_size - min_chunk_size) / kNumGearhashTables;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // a user defined `norm_level` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_level we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_level is 0 by
+  // default
+  int effective_bits = mask_bits - norm_level;
+
+  if (effective_bits < 1 || effective_bits > 63) {
+    throw ParquetException(
+        "The number of bits in the CDC mask must be between 1 and 63, got " +
+        std::to_string(effective_bits));
+  } else {
+    // create the mask by setting the top bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+}  // namespace
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int norm_level)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_level)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the

Review Comment:
   RapidJSON is always enabled when building Parquet.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086856970


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1689 @@
+// Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   Yes, that's what other tests do.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025051333


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+

Review Comment:
   Added using std::size() and also asserting on the overall size of the hash table to check that there are 8*256 entries overall.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025072705


##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <vector>
+
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;

Review Comment:
   Added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025196960


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();
+    auto min_length = ElementCount(min_chunk_size, byte_width, nullable);
+    auto max_length = ElementCount(max_chunk_size, byte_width, nullable);
+    AssertAllBetween(base_info.page_lengths, min_length, max_length);
+    AssertAllBetween(modified_info.page_lengths, min_length, max_length);
+  } else if (::arrow::is_base_binary_like(dtype->id()) && !enable_dictionary) {
+    AssertAllBetween(base_info.page_sizes, min_chunk_size, max_chunk_size);
+    AssertAllBetween(modified_info.page_sizes, min_chunk_size, max_chunk_size);
+  }
+}
+
+constexpr int64_t kMinChunkSize = 8 * 1024;
+constexpr int64_t kMaxChunkSize = 32 * 1024;
+constexpr int64_t kPartSize = 128 * 1024;
+constexpr int64_t kEditSize = 128;
+
+struct CaseConfig {
+  // Arrow data type to generate the testing data for
+  std::shared_ptr<::arrow::DataType> dtype;
+  // Whether the data type is nullable
+  bool is_nullable;
+  // Approximate number of bytes per record to calculate the number of elements to
+  // generate
+  size_t bytes_per_record;
+  // Data page version to use
+  ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1;
+};
+
+// Define PrintTo for MyStruct
+void PrintTo(const CaseConfig& param, std::ostream* os) {
+  *os << "{ " << param.dtype->ToString();
+  if (param.is_nullable) {
+    *os << " nullable";
+  }
+  *os << " }";
+}
+
+class TestCDC : public ::testing::Test {
+ public:
+  uint64_t GetMask(const ContentDefinedChunker& cdc) const { return cdc.GetMask(); }
+};
+
+TEST_F(TestCDC, RollingHashMaskCalculation) {
+  auto le = LevelInfo();
+  auto min_size = 256 * 1024;
+  auto max_size = 1024 * 1024;
+
+  auto cdc0 = ContentDefinedChunker(le, min_size, max_size, 0);
+  ASSERT_EQ(GetMask(cdc0), 0xFFFE000000000000);
+
+  auto cdc1 = ContentDefinedChunker(le, min_size, max_size, 1);
+  ASSERT_EQ(GetMask(cdc1), 0xFFFC000000000000);
+
+  auto cdc2 = ContentDefinedChunker(le, min_size, max_size, 2);
+  ASSERT_EQ(GetMask(cdc2), 0xFFF8000000000000);
+
+  auto cdc3 = ContentDefinedChunker(le, min_size, max_size, 3);
+  ASSERT_EQ(GetMask(cdc3), 0xFFF0000000000000);
+
+  auto cdc4 = ContentDefinedChunker(le, min_size, max_size, -1);
+  ASSERT_EQ(GetMask(cdc4), 0xFFFF000000000000);
+
+  // this is the smallest possible mask always matching, by using 8 hashtables
+  // we are going to have a match every 8 bytes; this is an unrealistic case
+  // but checking for the correctness of the mask calculation
+  auto cdc5 = ContentDefinedChunker(le, 0, 16, 0);
+  ASSERT_EQ(GetMask(cdc5), 0x0000000000000000);
+
+  auto cdc6 = ContentDefinedChunker(le, 0, 32, 1);
+  ASSERT_EQ(GetMask(cdc6), 0x0000000000000000);
+
+  auto cdc7 = ContentDefinedChunker(le, 0, 16, -1);
+  ASSERT_EQ(GetMask(cdc7), 0x8000000000000000);
+
+  // another unrealistic case, checking for the validation
+  auto cdc8 = ContentDefinedChunker(le, 128, 384, -60);
+  ASSERT_EQ(GetMask(cdc8), 0xFFFFFFFFFFFFFFFF);
+}
+
+TEST_F(TestCDC, WriteSingleColumnParquetFile) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  std::vector<int32_t> numbers = {1, 2, 3, 4, 5};
+  std::vector<uint8_t> valid_bits = {1, 0, 1, 0, 1};
+  EXPECT_THROW(
+      int_column_writer.WriteBatch(numbers.size(), nullptr, nullptr, numbers.data()),
+      ParquetException);
+  EXPECT_THROW(int_column_writer.WriteBatchSpaced(numbers.size(), nullptr, nullptr,
+                                                  valid_bits.data(), 0, numbers.data()),
+               ParquetException);
+}
+
+TEST_F(TestCDC, LastChunkDoesntTriggerAddDataPage) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()
+                   ->content_defined_chunking_options(kMinChunkSize, kMaxChunkSize, 0)
+                   ->disable_dictionary()
+                   ->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  ASSERT_OK_AND_ASSIGN(auto array, ::arrow::gen::Step()->Generate(8000));
+  auto arrow_props = default_arrow_writer_properties();
+  auto arrow_ctx = ArrowWriteContext(default_memory_pool(), arrow_props.get());
+
+  // Calling WriteArrow twice, we expect that the first call doesn't add a new data page
+  // at the end allowing subsequent calls to append to the same page
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+
+  int_column_writer.Close();
+  writer->Close();
+  ASSERT_OK_AND_ASSIGN(auto buffer, sink->Finish());
+
+  auto info = GetColumnParquetInfo(buffer);
+  ASSERT_EQ(info.size(), 1);
+
+  // AssertAllBetween allow the last chunk size to be smaller than the min_chunk_size
+  AssertAllBetween(info[0].page_sizes, kMinChunkSize, kMaxChunkSize);
+  AssertAllBetween(info[0].page_lengths, 3000, 5000);
+}
+
+TEST_F(TestCDC, ChunkSizeParameterValidation) {
+  // Test that constructor validates min/max chunk size parameters
+  auto li = LevelInfo();
+
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 256 * 1024, 1024 * 1024));
+
+  // with norm_factor=0 the difference between min and max chunk size must be
+  // at least 16
+  ASSERT_THROW(ContentDefinedChunker(li, 0, -1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1024, 512), ParquetException);
+
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16));
+  ASSERT_THROW(ContentDefinedChunker(li, -16, -16), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 16, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 32, 32), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 32, 48));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1024 * 1024, 2 * 1024 * 1024));
+  ASSERT_NO_THROW(
+      ContentDefinedChunker(li, 1024 * 1024 * 1024L, 2LL * 1024 * 1024 * 1024L));
+
+  // with norm_factor=1 the difference between min and max chunk size must be
+  // at least 64
+  ASSERT_THROW(ContentDefinedChunker(li, 1, -1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 32, 1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1, 33, 1));
+
+  // with norm_factor=2 the difference between min and max chunk size must be
+  // at least 128
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 63, 2), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 64, 2));
+
+  // with norm_factor=-1 the difference between min and max chunk size must be
+  // at least 8
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 7, -1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 8, -1));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16, -1));
+
+  // test the norm_factor extremes
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0, -68), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 1, -67));
+  ASSERT_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 59),
+               ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 58));
+}
+
+class TestCDCSingleRowGroup : public ::testing::TestWithParam<CaseConfig> {
+ protected:
+  // Column random table parts for testing
+  std::shared_ptr<Field> field_;
+  std::shared_ptr<Table> part1_, part2_, part3_, part4_, part5_, part6_, part7_;
+
+  void SetUp() override {
+    const auto& param = GetParam();
+    auto field_ = ::arrow::field("f0", param.dtype, param.is_nullable);
+    auto schema = ::arrow::schema({field_});
+
+    auto part_length = kPartSize / param.bytes_per_record;
+    auto edit_length = kEditSize / param.bytes_per_record;
+    ASSERT_OK_AND_ASSIGN(part1_, GenerateTable(schema, part_length, 0));
+    ASSERT_OK_AND_ASSIGN(part2_, GenerateTable(schema, edit_length, 1));
+    ASSERT_OK_AND_ASSIGN(part3_, GenerateTable(schema, part_length, part_length));

Review Comment:
   Added wherever GenerateTable() is being used.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1983523887


##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage system (CAS)
+/// which splits the bytes stream into content defined blobs. The CAS system will
+/// calculate a unique identifier for each blob, then store the blob in a key-value store.
+/// If the same blob is encountered again, the system can refer to the hash instead of
+/// physically storing the blob again. In the example above, the CAS system would store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks.
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data
+///   Deduplication
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+/// - Git is for Data (chunk size normalization used here is described in section 6.2.1):
+///   https://www.cidrdb.org/cidr2023/papers/p43-low.pdf
+class ContentDefinedChunker {
+ public:
+  /// Create a new ContentDefinedChunker instance
+  ///
+  /// @param level_info Information about definition and repetition levels
+  /// @param size_range Min/max chunk size as pair<min_size, max_size>, the chunker will
+  ///                   attempt to uniformly distribute the chunks between these extremes.
+  /// @param norm_factor Normalization factor to center the chunk size around the average
+  ///                    size more aggressively. By increasing the normalization factor,
+  ///                    probability of finding a chunk boundary increases.
+  ContentDefinedChunker(const LevelInfo& level_info,
+                        std::pair<uint64_t, uint64_t> size_range,
+                        uint8_t norm_factor = 0);

Review Comment:
   Updating.



##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage system (CAS)
+/// which splits the bytes stream into content defined blobs. The CAS system will
+/// calculate a unique identifier for each blob, then store the blob in a key-value store.
+/// If the same blob is encountered again, the system can refer to the hash instead of
+/// physically storing the blob again. In the example above, the CAS system would store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks.
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data
+///   Deduplication
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+/// - Git is for Data (chunk size normalization used here is described in section 6.2.1):
+///   https://www.cidrdb.org/cidr2023/papers/p43-low.pdf
+class ContentDefinedChunker {
+ public:
+  /// Create a new ContentDefinedChunker instance
+  ///
+  /// @param level_info Information about definition and repetition levels
+  /// @param size_range Min/max chunk size as pair<min_size, max_size>, the chunker will
+  ///                   attempt to uniformly distribute the chunks between these extremes.

Review Comment:
   Updating.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1993013786


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             int64_t min_size, int64_t max_size,
+                                             int8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(min_size),
+      max_size_(max_size),
+      hash_mask_(GetMask(min_size, max_size, norm_factor)) {
+  if (min_size_ < 0) {
+    throw ParquetException("min_size must be non-negative");
+  }
+  if (max_size_ < 0) {
+    throw ParquetException("max_size must be non-negative");
+  }
+  if (min_size_ > max_size_) {
+    throw ParquetException("min_size must be less than or equal to max_size");
+  }
+}
+
+void ContentDefinedChunker::Roll(const bool value) {
+  if (chunk_size_++ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+  has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+}
+
+template <int ByteWidth>
+void ContentDefinedChunker::Roll(const uint8_t* value) {
+  chunk_size_ += ByteWidth;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (size_t i = 0; i < ByteWidth; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T* value) {
+  return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+}
+
+void ContentDefinedChunker::Roll(const uint8_t* value, int64_t length) {
+  chunk_size_ += length;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (auto i = 0; i < length; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025762897


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);

Review Comment:
   These are rather fragile assertions, depending on the CDC parameters these change streaks can be longer, especially with smaller chunk sizes used in these test scenarios.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2026591682


##########
cpp/src/parquet/properties.h:
##########
@@ -245,6 +245,14 @@ class PARQUET_EXPORT ColumnProperties {
   bool page_index_enabled_;
 };
 
+struct CDCOptions {
+  int64_t min_chunk_size;
+  int64_t max_chunk_size;
+  int8_t norm_factor;
+};
+
+static constexpr CDCOptions kDefaultCdcOptions = CDCOptions{256 * 1024, 1024 * 1024, 0};

Review Comment:
   > Using `CDCOptions`, let me know if I should prefer the other.
   
   I have a slight preference for `CdcOptions`, but I don't mind really.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025137614


##########
cpp/src/parquet/properties.h:
##########
@@ -275,10 +285,60 @@ class PARQUET_EXPORT WriterProperties {
           page_checksum_enabled_(properties.page_checksum_enabled()),
           size_statistics_level_(properties.size_statistics_level()),
           sorting_columns_(properties.sorting_columns()),
-          default_column_properties_(properties.default_column_properties()) {}
+          default_column_properties_(properties.default_column_properties()),
+          content_defined_chunking_enabled_(
+              properties.content_defined_chunking_enabled()),
+          content_defined_chunking_options_(
+              properties.content_defined_chunking_options()) {}
 
     virtual ~Builder() {}
 
+    /// \brief EXPERIMENTAL: Use content-defined page chunking for all columns.
+    ///
+    /// Optimize parquet files for content addressable storage (CAS) systems by writing
+    /// data pages according to content-defined chunk boundaries. This allows for more
+    /// efficient deduplication of data across files, hence more efficient network
+    /// transfers and storage. The chunking is based on a rolling hash algorithm that
+    /// identifies chunk boundaries based on the actual content of the data.
+    Builder* enable_content_defined_chunking() {
+      content_defined_chunking_enabled_ = true;
+      return this;
+    }
+
+    /// \brief EXPERIMENTAL: Disable content-defined page chunking for all columns.
+    Builder* disable_content_defined_chunking() {
+      content_defined_chunking_enabled_ = false;
+      return this;
+    }
+
+    /// \brief EXPERIMENTAL: Specify content-defined chunking options.
+    ///
+    /// \param min_chunk_size Minimum chunk size in bytes, default 256 KiB
+    /// The rolling hash will not be updated until this size is reached for each chunk.
+    /// Note that all data sent through the hash function is counted towards the chunk
+    /// size, including definition and repetition levels if present.
+    /// \param max_chunk_size Maximum chunk size in bytes, default is 1024 KiB
+    /// The chunker will create a new chunk whenever the chunk size exceeds this value.
+    /// Note that the parquet writer has a related `pagesize` property that controls
+    /// the maximum size of a parquet data page after encoding. While setting
+    /// `pagesize` to a smaller value than `max_chunk_size` doesn't affect the
+    /// chunking effectiveness, it results in more small parquet data pages.
+    /// \param norm_factor Normalization factor to center the chunk size around the
+    /// average size more aggressively, default 0
+    /// Increasing the normalization factor increases the probability of finding a chunk,
+    /// improving the deduplication ratio, but also increasing the number of small chunks
+    /// resulting in many small parquet data pages. The default value provides a good
+    /// balance between deduplication ratio and fragmentation. Use norm_factor=1 or
+    /// norm_factor=2 to reach a higher deduplication ratio at the expense of
+    /// fragmentation.
+    Builder* content_defined_chunking_options(
+        int64_t min_chunk_size, int64_t max_chunk_size,
+        int8_t norm_factor = kDefaultCdcOptions.norm_factor) {

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025139649


##########
cpp/src/parquet/properties.h:
##########
@@ -245,6 +245,14 @@ class PARQUET_EXPORT ColumnProperties {
   bool page_index_enabled_;
 };
 
+struct CDCOptions {
+  int64_t min_chunk_size;
+  int64_t max_chunk_size;
+  int8_t norm_factor;

Review Comment:
   Using plain `int` and set default to 0.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027827840


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();
+    auto min_length = ElementCount(min_chunk_size, byte_width, nullable);
+    auto max_length = ElementCount(max_chunk_size, byte_width, nullable);
+    AssertAllBetween(base_info.page_lengths, min_length, max_length);
+    AssertAllBetween(modified_info.page_lengths, min_length, max_length);
+  } else if (::arrow::is_base_binary_like(dtype->id()) && !enable_dictionary) {
+    AssertAllBetween(base_info.page_sizes, min_chunk_size, max_chunk_size);
+    AssertAllBetween(modified_info.page_sizes, min_chunk_size, max_chunk_size);
+  }
+}
+
+constexpr int64_t kMinChunkSize = 8 * 1024;
+constexpr int64_t kMaxChunkSize = 32 * 1024;
+constexpr int64_t kPartSize = 128 * 1024;
+constexpr int64_t kEditSize = 128;
+
+struct CaseConfig {
+  // Arrow data type to generate the testing data for
+  std::shared_ptr<::arrow::DataType> dtype;
+  // Whether the data type is nullable
+  bool is_nullable;
+  // Approximate number of bytes per record to calculate the number of elements to
+  // generate
+  size_t bytes_per_record;
+  // Data page version to use
+  ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1;
+};
+
+// Define PrintTo for MyStruct
+void PrintTo(const CaseConfig& param, std::ostream* os) {
+  *os << "{ " << param.dtype->ToString();
+  if (param.is_nullable) {
+    *os << " nullable";
+  }
+  *os << " }";
+}
+
+class TestCDC : public ::testing::Test {
+ public:
+  uint64_t GetMask(const ContentDefinedChunker& cdc) const { return cdc.GetMask(); }
+};
+
+TEST_F(TestCDC, RollingHashMaskCalculation) {
+  auto le = LevelInfo();
+  auto min_size = 256 * 1024;
+  auto max_size = 1024 * 1024;
+
+  auto cdc0 = ContentDefinedChunker(le, min_size, max_size, 0);
+  ASSERT_EQ(GetMask(cdc0), 0xFFFE000000000000);
+
+  auto cdc1 = ContentDefinedChunker(le, min_size, max_size, 1);
+  ASSERT_EQ(GetMask(cdc1), 0xFFFC000000000000);
+
+  auto cdc2 = ContentDefinedChunker(le, min_size, max_size, 2);
+  ASSERT_EQ(GetMask(cdc2), 0xFFF8000000000000);
+
+  auto cdc3 = ContentDefinedChunker(le, min_size, max_size, 3);
+  ASSERT_EQ(GetMask(cdc3), 0xFFF0000000000000);
+
+  auto cdc4 = ContentDefinedChunker(le, min_size, max_size, -1);
+  ASSERT_EQ(GetMask(cdc4), 0xFFFF000000000000);
+
+  // this is the smallest possible mask always matching, by using 8 hashtables
+  // we are going to have a match every 8 bytes; this is an unrealistic case
+  // but checking for the correctness of the mask calculation
+  auto cdc5 = ContentDefinedChunker(le, 0, 16, 0);
+  ASSERT_EQ(GetMask(cdc5), 0x0000000000000000);
+
+  auto cdc6 = ContentDefinedChunker(le, 0, 32, 1);
+  ASSERT_EQ(GetMask(cdc6), 0x0000000000000000);
+
+  auto cdc7 = ContentDefinedChunker(le, 0, 16, -1);
+  ASSERT_EQ(GetMask(cdc7), 0x8000000000000000);
+
+  // another unrealistic case, checking for the validation
+  auto cdc8 = ContentDefinedChunker(le, 128, 384, -60);
+  ASSERT_EQ(GetMask(cdc8), 0xFFFFFFFFFFFFFFFF);
+}
+
+TEST_F(TestCDC, WriteSingleColumnParquetFile) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  std::vector<int32_t> numbers = {1, 2, 3, 4, 5};
+  std::vector<uint8_t> valid_bits = {1, 0, 1, 0, 1};
+  EXPECT_THROW(
+      int_column_writer.WriteBatch(numbers.size(), nullptr, nullptr, numbers.data()),
+      ParquetException);
+  EXPECT_THROW(int_column_writer.WriteBatchSpaced(numbers.size(), nullptr, nullptr,
+                                                  valid_bits.data(), 0, numbers.data()),
+               ParquetException);
+}
+
+TEST_F(TestCDC, LastChunkDoesntTriggerAddDataPage) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()
+                   ->content_defined_chunking_options(kMinChunkSize, kMaxChunkSize, 0)
+                   ->disable_dictionary()
+                   ->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  ASSERT_OK_AND_ASSIGN(auto array, ::arrow::gen::Step()->Generate(8000));
+  auto arrow_props = default_arrow_writer_properties();
+  auto arrow_ctx = ArrowWriteContext(default_memory_pool(), arrow_props.get());
+
+  // Calling WriteArrow twice, we expect that the first call doesn't add a new data page
+  // at the end allowing subsequent calls to append to the same page
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+
+  int_column_writer.Close();
+  writer->Close();
+  ASSERT_OK_AND_ASSIGN(auto buffer, sink->Finish());
+
+  auto info = GetColumnParquetInfo(buffer);
+  ASSERT_EQ(info.size(), 1);
+
+  // AssertAllBetween allow the last chunk size to be smaller than the min_chunk_size
+  AssertAllBetween(info[0].page_sizes, kMinChunkSize, kMaxChunkSize);
+  AssertAllBetween(info[0].page_lengths, 3000, 5000);
+}
+
+TEST_F(TestCDC, ChunkSizeParameterValidation) {
+  // Test that constructor validates min/max chunk size parameters
+  auto li = LevelInfo();
+
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 256 * 1024, 1024 * 1024));
+
+  // with norm_factor=0 the difference between min and max chunk size must be
+  // at least 16
+  ASSERT_THROW(ContentDefinedChunker(li, 0, -1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1024, 512), ParquetException);
+
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16));
+  ASSERT_THROW(ContentDefinedChunker(li, -16, -16), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 16, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 32, 32), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 32, 48));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1024 * 1024, 2 * 1024 * 1024));
+  ASSERT_NO_THROW(
+      ContentDefinedChunker(li, 1024 * 1024 * 1024L, 2LL * 1024 * 1024 * 1024L));
+
+  // with norm_factor=1 the difference between min and max chunk size must be
+  // at least 64
+  ASSERT_THROW(ContentDefinedChunker(li, 1, -1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 32, 1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1, 33, 1));
+
+  // with norm_factor=2 the difference between min and max chunk size must be
+  // at least 128
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 63, 2), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 64, 2));
+
+  // with norm_factor=-1 the difference between min and max chunk size must be
+  // at least 8
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 7, -1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 8, -1));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16, -1));
+
+  // test the norm_factor extremes
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0, -68), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 1, -67));
+  ASSERT_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 59),
+               ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 58));
+}
+
+class TestCDCSingleRowGroup : public ::testing::TestWithParam<CaseConfig> {
+ protected:
+  // Column random table parts for testing
+  std::shared_ptr<Field> field_;
+  std::shared_ptr<Table> part1_, part2_, part3_, part4_, part5_, part6_, part7_;
+
+  void SetUp() override {
+    const auto& param = GetParam();
+    auto field_ = ::arrow::field("f0", param.dtype, param.is_nullable);
+    auto schema = ::arrow::schema({field_});
+
+    auto part_length = kPartSize / param.bytes_per_record;
+    auto edit_length = kEditSize / param.bytes_per_record;
+    ASSERT_OK_AND_ASSIGN(part1_, GenerateTable(schema, part_length, 0));
+    ASSERT_OK_AND_ASSIGN(part2_, GenerateTable(schema, edit_length, 1));
+    ASSERT_OK_AND_ASSIGN(part3_, GenerateTable(schema, part_length, part_length));
+    ASSERT_OK_AND_ASSIGN(part4_, GenerateTable(schema, edit_length, 2));
+    ASSERT_OK_AND_ASSIGN(part5_, GenerateTable(schema, part_length, 2 * part_length));
+    ASSERT_OK_AND_ASSIGN(part6_, GenerateTable(schema, edit_length, 3));
+    ASSERT_OK_AND_ASSIGN(part7_, GenerateTable(schema, edit_length, 4));
+  }
+};
+
+TEST_P(TestCDCSingleRowGroup, DeleteOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }

Review Comment:
   I made it more robust adding the number of `null` / `empty list` elements to the number of values, added a comment as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027907788


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int8_t norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes
+  int64_t target_size = avg_chunk_size - min_chunk_size;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // 3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  if (norm_factor < -3 || norm_factor > 3) {
+    ARROW_LOG(WARNING) << "norm_factor=" << norm_factor
+                       << " is outside the recommended range [-3, 3]";
+  }
+
+  int mask_adjustement = 3 + norm_factor;
+  int effective_bits = mask_bits - mask_adjustement;
+  if (effective_bits == 0) {
+    return 0;
+  } else if (effective_bits > 64) {
+    throw ParquetException("The number of bits in the mask cannot exceed 64, got " +
+                           std::to_string(effective_bits));
+  } else if (effective_bits < 0) {
+    throw ParquetException(
+        "The difference between min_chunk_size=" + std::to_string(min_chunk_size) +
+        " and max_chunk_size=" + std::to_string(max_chunk_size) +
+        " is too small for the given norm_factor=" + std::to_string(norm_factor) +
+        ", either increase the size range or decrease the norm_factor.");
+  } else {
+    // create the mask by setting the top mask_bits bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int8_t norm_factor)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_factor)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(const bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+    has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+  }
+
+  template <int ByteWidth>
+  void Roll(const uint8_t* value) {
+    // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+    // true if the hash matches the mask.
+
+    chunk_size_ += ByteWidth;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (size_t i = 0; i < ByteWidth; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  template <typename T>
+  void Roll(const T* value) {
+    return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+  }
+
+  void Roll(const uint8_t* value, int64_t length) {
+    // Update the rolling hash with a binary-like value, set has_matched_ to true if the
+    // hash matches the mask.
+
+    chunk_size_ += length;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (auto i = 0; i < length; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  bool NeedNewChunk() {
+    // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+    // set to true if we encountered a match since the last NeedNewChunk() call
+    if (ARROW_PREDICT_FALSE(has_matched_)) {
+      has_matched_ = false;
+      // in order to have a normal distribution of chunk sizes, we only create a new chunk
+      // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+      // different gearhash table (gearhash's chunk size has geometric distribution, and
+      // we use central limit theorem to approximate normal distribution, see
+      // section 6.2.1 in paper https://www.cidrdb.org/cidr2023/papers/p43-low.pdf)
+      if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+        nth_run_ = 0;
+        chunk_size_ = 0;
+        return true;
+      }
+    }
+    if (ARROW_PREDICT_FALSE(chunk_size_ >= max_chunk_size_)) {
+      // we have a hard limit on the maximum chunk size, note that we don't reset the
+      // rolling hash state here, so the next NeedNewChunk() call will continue from the
+      // current state
+      chunk_size_ = 0;
+      return true;
+    }
+    return false;
+  }
+
+  void ValidateChunks(const std::vector<Chunk>& chunks, int64_t num_levels) {
+    // chunks must be non-empty and monotonic increasing
+    DCHECK(!chunks.empty());
+
+    // the first chunk must start at the first level
+    auto first_chunk = chunks.front();
+    DCHECK_EQ(first_chunk.level_offset, 0);
+    DCHECK_EQ(first_chunk.value_offset, 0);
+
+    // the following chunks must be contiguous, non-overlapping and monotonically
+    // increasing
+    auto sum_levels = first_chunk.levels_to_write;
+    for (size_t i = 1; i < chunks.size(); ++i) {
+      auto chunk = chunks[i];
+      auto prev_chunk = chunks[i - 1];
+      DCHECK_GT(chunk.levels_to_write, 0);
+      DCHECK_GE(chunk.value_offset, prev_chunk.value_offset);
+      DCHECK_EQ(chunk.level_offset, prev_chunk.level_offset + prev_chunk.levels_to_write);
+      sum_levels += chunk.levels_to_write;
+    }
+    DCHECK_EQ(sum_levels, num_levels);
+
+    // the last chunk must end at the last level
+    auto last_chunk = chunks.back();
+    DCHECK_EQ(last_chunk.level_offset + last_chunk.levels_to_write, num_levels);
+  }
+
+  template <typename RollFunc>
+  std::vector<Chunk> Calculate(const int16_t* def_levels, const int16_t* rep_levels,
+                               int64_t num_levels, const RollFunc& RollValue) {
+    // Calculate the chunk boundaries for typed Arrow arrays.
+    std::vector<Chunk> chunks;
+    int64_t offset;
+    int64_t prev_offset = 0;
+    int64_t prev_value_offset = 0;
+    bool has_def_levels = level_info_.def_level > 0;
+    bool has_rep_levels = level_info_.rep_level > 0;
+
+    if (!has_rep_levels && !has_def_levels) {
+      // fastest path for non-nested non-null data
+      for (offset = 0; offset < num_levels; ++offset) {
+        RollValue(offset);
+        if (NeedNewChunk()) {
+          chunks.push_back({prev_offset, prev_offset, offset - prev_offset});
+          prev_offset = offset;
+        }
+      }
+      // set the previous value offset to add the last chunk
+      prev_value_offset = prev_offset;
+    } else if (!has_rep_levels) {
+      // non-nested data with nulls
+      int16_t def_level;
+      for (int64_t offset = 0; offset < num_levels; ++offset) {
+        def_level = def_levels[offset];
+
+        Roll(&def_level);
+        if (def_level == level_info_.def_level) {
+          RollValue(offset);
+        }
+        if (NeedNewChunk()) {
+          chunks.push_back({prev_offset, prev_offset, offset - prev_offset});
+          prev_offset = offset;
+        }
+      }
+      // set the previous value offset to add the last chunk
+      prev_value_offset = prev_offset;
+    } else {
+      // nested data with nulls
+      int16_t def_level;
+      int16_t rep_level;
+      int64_t value_offset = 0;
+
+      for (offset = 0; offset < num_levels; ++offset) {
+        def_level = def_levels[offset];
+        rep_level = rep_levels[offset];
+
+        Roll(&def_level);
+        Roll(&rep_level);
+        if (def_level == level_info_.def_level) {
+          RollValue(value_offset);
+        }
+
+        if ((rep_level == 0) && NeedNewChunk()) {
+          // if we are at a record boundary and need a new chunk, we create a new chunk
+          auto levels_to_write = offset - prev_offset;
+          if (levels_to_write > 0) {
+            chunks.push_back({prev_offset, prev_value_offset, levels_to_write});
+            prev_offset = offset;
+            prev_value_offset = value_offset;
+          }
+        }
+        if (def_level >= level_info_.repeated_ancestor_def_level) {
+          // we only increment the value offset if we have a leaf value
+          ++value_offset;
+        }
+      }
+    }
+
+    // add the last chunk if we have any levels left
+    if (prev_offset < num_levels) {
+      chunks.push_back({prev_offset, prev_value_offset, num_levels - prev_offset});
+    }
+#ifndef NDEBUG
+    ValidateChunks(chunks, num_levels);
+#endif
+
+    return chunks;
+  }
+
+  template <int kByteWidth>
+  std::vector<Chunk> CalculateFixedWidth(const int16_t* def_levels,
+                                         const int16_t* rep_levels, int64_t num_levels,
+                                         const ::arrow::Array& values) {
+    const uint8_t* raw_values =
+        values.data()->GetValues<uint8_t>(1, 0) + values.offset() * kByteWidth;
+    return Calculate(def_levels, rep_levels, num_levels, [&](int64_t i) {
+      return Roll<kByteWidth>(&raw_values[i * kByteWidth]);
+    });
+  }
+
+  template <typename ArrayType>
+  std::vector<Chunk> CalculateBinaryLike(const int16_t* def_levels,
+                                         const int16_t* rep_levels, int64_t num_levels,
+                                         const ::arrow::Array& values) {
+    const auto& array = checked_cast<const ArrayType&>(values);
+    const uint8_t* value;
+    typename ArrayType::offset_type length;
+    return Calculate(def_levels, rep_levels, num_levels, [&](int64_t i) {
+      value = array.GetValue(i, &length);
+      Roll(value, length);
+    });

Review Comment:
   Well, I don't have benchmarks yet so I assume either one is just as good. I preferred `GetValue` to spare intermediate construction of `string_view` (unless the compiler is able to optimize it away). Removing the capture.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1976493429


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,773 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {

Review Comment:
   Renamed it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1973983358


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+bool ContentDefinedChunker::Check() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last Check() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling
+    // hash state here, so the next Check() call will continue from the current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename T>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const T& leaf_array) {
+  std::vector<Chunk> result;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(def_levels[offset]);
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else {
+    // nested data with nulls
+    bool has_leaf_value;
+    bool is_record_boundary;
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t level_offset = 0;
+    int64_t value_offset = 0;
+    int64_t record_level_offset = 0;
+    int64_t record_value_offset = 0;
+
+    while (level_offset < num_levels) {

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1973231117


##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1332,13 +1337,38 @@ class TypedColumnWriterImpl : public ColumnWriterImpl, public TypedColumnWriter<
       bits_buffer_->ZeroPadding();
     }
 
-    if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
-      return WriteArrowDictionary(def_levels, rep_levels, num_levels, leaf_array, ctx,
-                                  maybe_parent_nulls);
+    if (this->properties_->cdc_enabled()) {
+      ARROW_ASSIGN_OR_RAISE(auto boundaries,
+                            content_defined_chunker_.GetBoundaries(
+                                def_levels, rep_levels, num_levels, leaf_array));
+      for (auto chunk : boundaries) {
+        auto chunk_array = leaf_array.Slice(chunk.value_offset);
+        auto chunk_def_levels = AddIfNotNull(def_levels, chunk.level_offset);
+        auto chunk_rep_levels = AddIfNotNull(rep_levels, chunk.level_offset);
+        if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
+          ARROW_CHECK_OK(WriteArrowDictionary(chunk_def_levels, chunk_rep_levels,
+                                              chunk.levels_to_write, *chunk_array, ctx,
+                                              maybe_parent_nulls));
+        } else {
+          ARROW_CHECK_OK(WriteArrowDense(chunk_def_levels, chunk_rep_levels,
+                                         chunk.levels_to_write, *chunk_array, ctx,
+                                         maybe_parent_nulls));
+        }
+        if (num_buffered_values_ > 0) {
+          AddDataPage();
+        }

Review Comment:
   Yes, the data pages need to be separated according to the chunks. Going to add a comment about this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kou (via GitHub)" <gi...@apache.org>]

kou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1972862484


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated

Review Comment:
   Hmm. Should we use "gearhash" not "gear hash"?



##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+bool ContentDefinedChunker::Check() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last Check() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling
+    // hash state here, so the next Check() call will continue from the current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename T>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const T& leaf_array) {
+  std::vector<Chunk> result;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(def_levels[offset]);
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else {
+    // nested data with nulls
+    bool has_leaf_value;
+    bool is_record_boundary;
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t level_offset = 0;
+    int64_t value_offset = 0;
+    int64_t record_level_offset = 0;
+    int64_t record_value_offset = 0;
+
+    while (level_offset < num_levels) {
+      def_level = def_levels[level_offset];
+      rep_level = rep_levels[level_offset];
+
+      has_leaf_value = def_level >= level_info_.repeated_ancestor_def_level;
+      is_record_boundary = rep_level == 0;
+
+      Roll(def_level);
+      Roll(rep_level);
+      if (has_leaf_value) {
+        Roll(leaf_array.GetView(value_offset));
+      }
+
+      if (is_record_boundary && Check()) {
+        auto levels_to_write = level_offset - record_level_offset;
+        if (levels_to_write > 0) {
+          result.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+          record_level_offset = level_offset;
+          record_value_offset = value_offset;
+        }
+      }
+
+      ++level_offset;
+      if (has_leaf_value) {
+        ++value_offset;
+      }
+    }
+
+    auto levels_to_write = num_levels - record_level_offset;
+    if (levels_to_write > 0) {
+      result.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+    }
+  }
+
+  return result;
+}
+
+#define PRIMITIVE_CASE(TYPE_ID, ArrowType)               \
+  case ::arrow::Type::TYPE_ID:                           \
+    return Calculate(def_levels, rep_levels, num_levels, \
+                     static_cast<const ::arrow::ArrowType##Array&>(values));
+
+const ::arrow::Result<std::vector<Chunk>> ContentDefinedChunker::GetBoundaries(
+    const int16_t* def_levels, const int16_t* rep_levels, int64_t num_levels,
+    const ::arrow::Array& values) {
+  auto type_id = values.type()->id();
+  switch (type_id) {

Review Comment:
   Can we reuse `VisitArrayInline()`?
   https://github.com/apache/arrow/blob/main/cpp/src/arrow/visit_array_inline.h
   
   



##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {

Review Comment:
   Could you add the Python script to this repository?
   We may want to change the gear hash tables later.



##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }

Review Comment:
   I'm not sure whether this is a critical pass or not but we may be able to eliminate this branch by `has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) ==0)`. This may improve performance.



##########
python/pyarrow/_parquet.pyx:
##########
@@ -1996,6 +2001,18 @@ cdef shared_ptr[WriterProperties] _create_writer_properties(
     if dictionary_pagesize_limit is not None:
         props.dictionary_pagesize_limit(dictionary_pagesize_limit)
 
+    # content defined chunking
+    if cdc is False:
+        props.disable_cdc()
+    elif cdc is True:
+        props.enable_cdc()

Review Comment:
   ```suggestion
       if cdc:
           props.enable_cdc()
       else:
           props.disable_cdc()
   ```



##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1332,13 +1337,38 @@ class TypedColumnWriterImpl : public ColumnWriterImpl, public TypedColumnWriter<
       bits_buffer_->ZeroPadding();
     }
 
-    if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
-      return WriteArrowDictionary(def_levels, rep_levels, num_levels, leaf_array, ctx,
-                                  maybe_parent_nulls);
+    if (this->properties_->cdc_enabled()) {
+      ARROW_ASSIGN_OR_RAISE(auto boundaries,
+                            content_defined_chunker_.GetBoundaries(
+                                def_levels, rep_levels, num_levels, leaf_array));
+      for (auto chunk : boundaries) {
+        auto chunk_array = leaf_array.Slice(chunk.value_offset);
+        auto chunk_def_levels = AddIfNotNull(def_levels, chunk.level_offset);
+        auto chunk_rep_levels = AddIfNotNull(rep_levels, chunk.level_offset);
+        if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
+          ARROW_CHECK_OK(WriteArrowDictionary(chunk_def_levels, chunk_rep_levels,
+                                              chunk.levels_to_write, *chunk_array, ctx,
+                                              maybe_parent_nulls));
+        } else {
+          ARROW_CHECK_OK(WriteArrowDense(chunk_def_levels, chunk_rep_levels,
+                                         chunk.levels_to_write, *chunk_array, ctx,
+                                         maybe_parent_nulls));
+        }
+        if (num_buffered_values_ > 0) {
+          AddDataPage();
+        }

Review Comment:
   Do we need to do this manually?
   It seems that `WriteArrowDictionary()`/`WriteArrowDense()` do this internally. 



##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1332,13 +1337,38 @@ class TypedColumnWriterImpl : public ColumnWriterImpl, public TypedColumnWriter<
       bits_buffer_->ZeroPadding();
     }
 
-    if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
-      return WriteArrowDictionary(def_levels, rep_levels, num_levels, leaf_array, ctx,
-                                  maybe_parent_nulls);
+    if (this->properties_->cdc_enabled()) {
+      ARROW_ASSIGN_OR_RAISE(auto boundaries,
+                            content_defined_chunker_.GetBoundaries(
+                                def_levels, rep_levels, num_levels, leaf_array));
+      for (auto chunk : boundaries) {
+        auto chunk_array = leaf_array.Slice(chunk.value_offset);
+        auto chunk_def_levels = AddIfNotNull(def_levels, chunk.level_offset);
+        auto chunk_rep_levels = AddIfNotNull(rep_levels, chunk.level_offset);
+        if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
+          ARROW_CHECK_OK(WriteArrowDictionary(chunk_def_levels, chunk_rep_levels,
+                                              chunk.levels_to_write, *chunk_array, ctx,
+                                              maybe_parent_nulls));
+        } else {
+          ARROW_CHECK_OK(WriteArrowDense(chunk_def_levels, chunk_rep_levels,
+                                         chunk.levels_to_write, *chunk_array, ctx,
+                                         maybe_parent_nulls));
+        }
+        if (num_buffered_values_ > 0) {
+          AddDataPage();
+        }
+      }
+      return Status::OK();
     } else {
-      return WriteArrowDense(def_levels, rep_levels, num_levels, leaf_array, ctx,
-                             maybe_parent_nulls);
+      if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
+        return WriteArrowDictionary(def_levels, rep_levels, num_levels, leaf_array, ctx,
+                                    maybe_parent_nulls);
+      } else {
+        return WriteArrowDense(def_levels, rep_levels, num_levels, leaf_array, ctx,
+                               maybe_parent_nulls);
+      }

Review Comment:
   This change isn't related to CDC, right?



##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+

Review Comment:
   Can we add a comment that refers an URL that explains the algorithm used by this implementation?



##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1332,13 +1337,38 @@ class TypedColumnWriterImpl : public ColumnWriterImpl, public TypedColumnWriter<
       bits_buffer_->ZeroPadding();
     }
 
-    if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
-      return WriteArrowDictionary(def_levels, rep_levels, num_levels, leaf_array, ctx,
-                                  maybe_parent_nulls);
+    if (this->properties_->cdc_enabled()) {

Review Comment:
   ```suggestion
       if (properties_->cdc_enabled()) {
   ```



##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1332,13 +1337,38 @@ class TypedColumnWriterImpl : public ColumnWriterImpl, public TypedColumnWriter<
       bits_buffer_->ZeroPadding();
     }
 
-    if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
-      return WriteArrowDictionary(def_levels, rep_levels, num_levels, leaf_array, ctx,
-                                  maybe_parent_nulls);
+    if (this->properties_->cdc_enabled()) {
+      ARROW_ASSIGN_OR_RAISE(auto boundaries,
+                            content_defined_chunker_.GetBoundaries(
+                                def_levels, rep_levels, num_levels, leaf_array));
+      for (auto chunk : boundaries) {
+        auto chunk_array = leaf_array.Slice(chunk.value_offset);

Review Comment:
   It seems that we need to specify not only offset but also length. If we don't specify length, we may use duplicated data when `leaf_array` is split to 3 or more chunks.



##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+bool ContentDefinedChunker::Check() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last Check() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling
+    // hash state here, so the next Check() call will continue from the current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename T>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const T& leaf_array) {
+  std::vector<Chunk> result;

Review Comment:
   How about renaming to `chunks` for easy to read?
   
   ```suggestion
     std::vector<Chunk> chunks;
   ```



##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }

Review Comment:
   Hmm. I think that we need to update `rolling_hash_` in this case too.
   
   I think that we can use this optimization only when total of all target data size is less than `min_size_`.
   
   



##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+bool ContentDefinedChunker::Check() {

Review Comment:
   Can we use better name? `NeedNewChunk()`?



##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for

Review Comment:
   ```suggestion
     // we skip calculating gear hash for the first `min_size` bytes, so we are looking for
   ```



##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+bool ContentDefinedChunker::Check() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last Check() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling
+    // hash state here, so the next Check() call will continue from the current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename T>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const T& leaf_array) {
+  std::vector<Chunk> result;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(def_levels[offset]);
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else {
+    // nested data with nulls
+    bool has_leaf_value;
+    bool is_record_boundary;
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t level_offset = 0;
+    int64_t value_offset = 0;
+    int64_t record_level_offset = 0;
+    int64_t record_value_offset = 0;
+
+    while (level_offset < num_levels) {

Review Comment:
   `for (int64_t level_offset = 0; level_offset < num_levels; ++level_offset)` may be better.



##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+bool ContentDefinedChunker::Check() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last Check() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {

Review Comment:
   It seems that some chunks returned by `Calculate()` may be bigger than `max_size_`. We may need to check this in `Roll()` or before calling `Role()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1998840037


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             int64_t min_size, int64_t max_size,
+                                             int8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(min_size),
+      max_size_(max_size),
+      hash_mask_(GetMask(min_size, max_size, norm_factor)) {
+  if (min_size_ < 0) {
+    throw ParquetException("min_size must be non-negative");
+  }
+  if (max_size_ < 0) {
+    throw ParquetException("max_size must be non-negative");
+  }
+  if (min_size_ > max_size_) {
+    throw ParquetException("min_size must be less than or equal to max_size");
+  }
+}
+
+void ContentDefinedChunker::Roll(const bool value) {
+  if (chunk_size_++ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+  has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+}
+
+template <int ByteWidth>
+void ContentDefinedChunker::Roll(const uint8_t* value) {
+  chunk_size_ += ByteWidth;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (size_t i = 0; i < ByteWidth; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T* value) {
+  return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+}
+
+void ContentDefinedChunker::Roll(const uint8_t* value, int64_t length) {
+  chunk_size_ += length;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (auto i = 0; i < length; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, note that we don't reset the
+    // rolling hash state here, so the next NeedNewChunk() call will continue from the
+    // current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename RollFunc>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const RollFunc& RollValue) {
+  std::vector<Chunk> chunks;
+  int64_t offset;
+  int64_t prev_offset = 0;
+  int64_t prev_value_offset = 0;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    for (offset = 0; offset < num_levels; ++offset) {
+      RollValue(offset);
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int16_t def_level;
+    for (int64_t offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+
+      Roll(&def_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(offset);
+      }
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else {
+    // nested data with nulls
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t value_offset = 0;
+
+    for (offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+      rep_level = rep_levels[offset];
+
+      Roll(&def_level);
+      Roll(&rep_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(value_offset);
+      }
+
+      if ((rep_level == 0) && NeedNewChunk()) {
+        // if we are at a record boundary and need a new chunk, we create a new chunk
+        auto levels_to_write = offset - prev_offset;
+        if (levels_to_write > 0) {
+          chunks.emplace_back(prev_offset, prev_value_offset, levels_to_write);
+          prev_offset = offset;
+          prev_value_offset = value_offset;
+        }
+      }
+      if (def_level >= level_info_.repeated_ancestor_def_level) {
+        // we only increment the value offset if we have a leaf value
+        ++value_offset;
+      }
+    }
+  }
+
+  // add the last chunk if we have any levels left
+  if (prev_offset < num_levels) {
+    chunks.emplace_back(prev_offset, prev_value_offset, num_levels - prev_offset);
+  }
+  return chunks;
+}
+
+#define FIXED_WIDTH_CASE(ByteWidth)                                       \
+  {                                                                       \
+    const auto raw_values = values.data()->GetValues<uint8_t>(1);         \
+    return Calculate(def_levels, rep_levels, num_levels, [&](int64_t i) { \
+      return Roll<ByteWidth>(raw_values + i * ByteWidth);                 \
+    });                                                                   \
+  }
+
+#define BINARY_LIKE_CASE(ArrayType)                                       \
+  {                                                                       \
+    const auto& array = static_cast<const ArrayType&>(values);            \
+    const uint8_t* value;                                                 \
+    ArrayType::offset_type length;                                        \
+    return Calculate(def_levels, rep_levels, num_levels, [&](int64_t i) { \
+      value = array.GetValue(i, &length);                                 \
+      Roll(value, length);                                                \
+    });                                                                   \
+  }
+
+const std::vector<Chunk> ContentDefinedChunker::GetBoundaries(
+    const int16_t* def_levels, const int16_t* rep_levels, int64_t num_levels,
+    const ::arrow::Array& values) {
+  auto type_id = values.type()->id();
+  switch (type_id) {

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2005288118


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, uint64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal128Builder, ::arrow::Decimal128(val % max_value))
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal256Builder, ::arrow::Decimal256(val % max_value))
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      GENERATE_CASE_BODY(::arrow::FixedSizeBinaryBuilder,
+                         std::string("bin_") + std::to_string(val).substr(0, size - 4))
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, uint64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(const std::shared_ptr<Table>& table,
+                                                   uint64_t min_chunk_size,
+                                                   uint64_t max_chunk_size,
+                                                   bool enable_dictionary = false,
+
+                                                   int64_t row_group_size = 1024 * 1024) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_cdc()
+      ->cdc_size_range(min_chunk_size, max_chunk_size)
+      ->cdc_norm_factor(0);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }

Review Comment:
   Updated, dictionary fallback is tested except for boolean. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2694465805

   @github-actions crossbow submit wheel-*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1985543223


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, uint64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal128Builder, ::arrow::Decimal128(val % max_value))
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal256Builder, ::arrow::Decimal256(val % max_value))
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      GENERATE_CASE_BODY(::arrow::FixedSizeBinaryBuilder,
+                         std::string("bin_") + std::to_string(val).substr(0, size - 4))
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, uint64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(const std::shared_ptr<Table>& table,
+                                                   uint64_t min_chunk_size,
+                                                   uint64_t max_chunk_size,
+                                                   bool enable_dictionary = false,
+
+                                                   int64_t row_group_size = 1024 * 1024) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_cdc()
+      ->cdc_size_range(min_chunk_size, max_chunk_size)
+      ->cdc_norm_factor(0);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+struct PageSizes {
+  std::vector<uint64_t> lengths;
+  std::vector<uint64_t> sizes;
+};
+
+PageSizes GetColumnPageSizes(const std::shared_ptr<Buffer>& data, int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  PageSizes result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        result.sizes.push_back(data_page->size());
+        result.lengths.push_back(data_page->num_values());
+      }
+    }
+  }
+
+  return result;
+}
+
+Result<PageSizes> WriteAndGetPageSizes(const std::shared_ptr<Table>& table,
+                                       uint64_t min_chunk_size, uint64_t max_chunk_size,
+                                       bool enable_dictionary = false,
+                                       int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnPageSizes(buffer, column_index);
+}
+
+void AssertAllBetween(const std::vector<uint64_t>& values, uint64_t min, uint64_t max,
+                      bool expect_dictionary_fallback = false) {
+  // expect the last chunk since it is not guaranteed to be within the range
+  if (expect_dictionary_fallback) {
+    // if dictionary encoding is enabled, the writer can fallback to plain
+    // encoding splitting within a content defined chunk, so we can't
+    // guarantee that all chunks are within the range in this case, but we
+    // know that there can be at most 2 pages smaller than the min_chunk_size
+    size_t smaller_count = 0;
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      if (values[i] < min) {
+        smaller_count++;
+      } else {
+        ASSERT_LE(values[i], max);
+      }
+    }
+    ASSERT_LE(smaller_count, 2);
+  } else {
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      ASSERT_GE(values[i], min);
+      ASSERT_LE(values[i], max);
+    }
+  }
+  ASSERT_LE(values.back(), max);
+}
+
+std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>> FindDifferences(
+    const std::vector<uint64_t>& first, const std::vector<uint64_t>& second) {
+  // Compute LCS table.
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack to get common indices.
+  std::vector<std::pair<size_t, size_t>> common;

Review Comment:
   Extended the comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027823562


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();
+    auto min_length = ElementCount(min_chunk_size, byte_width, nullable);
+    auto max_length = ElementCount(max_chunk_size, byte_width, nullable);
+    AssertAllBetween(base_info.page_lengths, min_length, max_length);
+    AssertAllBetween(modified_info.page_lengths, min_length, max_length);
+  } else if (::arrow::is_base_binary_like(dtype->id()) && !enable_dictionary) {
+    AssertAllBetween(base_info.page_sizes, min_chunk_size, max_chunk_size);
+    AssertAllBetween(modified_info.page_sizes, min_chunk_size, max_chunk_size);
+  }
+}
+
+constexpr int64_t kMinChunkSize = 8 * 1024;
+constexpr int64_t kMaxChunkSize = 32 * 1024;
+constexpr int64_t kPartSize = 128 * 1024;
+constexpr int64_t kEditSize = 128;
+
+struct CaseConfig {
+  // Arrow data type to generate the testing data for
+  std::shared_ptr<::arrow::DataType> dtype;
+  // Whether the data type is nullable
+  bool is_nullable;
+  // Approximate number of bytes per record to calculate the number of elements to
+  // generate
+  size_t bytes_per_record;
+  // Data page version to use
+  ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1;
+};
+
+// Define PrintTo for MyStruct
+void PrintTo(const CaseConfig& param, std::ostream* os) {
+  *os << "{ " << param.dtype->ToString();
+  if (param.is_nullable) {
+    *os << " nullable";
+  }
+  *os << " }";
+}
+
+class TestCDC : public ::testing::Test {
+ public:
+  uint64_t GetMask(const ContentDefinedChunker& cdc) const { return cdc.GetMask(); }
+};
+
+TEST_F(TestCDC, RollingHashMaskCalculation) {
+  auto le = LevelInfo();
+  auto min_size = 256 * 1024;
+  auto max_size = 1024 * 1024;
+
+  auto cdc0 = ContentDefinedChunker(le, min_size, max_size, 0);
+  ASSERT_EQ(GetMask(cdc0), 0xFFFE000000000000);
+
+  auto cdc1 = ContentDefinedChunker(le, min_size, max_size, 1);
+  ASSERT_EQ(GetMask(cdc1), 0xFFFC000000000000);
+
+  auto cdc2 = ContentDefinedChunker(le, min_size, max_size, 2);
+  ASSERT_EQ(GetMask(cdc2), 0xFFF8000000000000);
+
+  auto cdc3 = ContentDefinedChunker(le, min_size, max_size, 3);
+  ASSERT_EQ(GetMask(cdc3), 0xFFF0000000000000);
+
+  auto cdc4 = ContentDefinedChunker(le, min_size, max_size, -1);
+  ASSERT_EQ(GetMask(cdc4), 0xFFFF000000000000);
+
+  // this is the smallest possible mask always matching, by using 8 hashtables
+  // we are going to have a match every 8 bytes; this is an unrealistic case
+  // but checking for the correctness of the mask calculation
+  auto cdc5 = ContentDefinedChunker(le, 0, 16, 0);
+  ASSERT_EQ(GetMask(cdc5), 0x0000000000000000);
+
+  auto cdc6 = ContentDefinedChunker(le, 0, 32, 1);
+  ASSERT_EQ(GetMask(cdc6), 0x0000000000000000);
+
+  auto cdc7 = ContentDefinedChunker(le, 0, 16, -1);
+  ASSERT_EQ(GetMask(cdc7), 0x8000000000000000);
+
+  // another unrealistic case, checking for the validation
+  auto cdc8 = ContentDefinedChunker(le, 128, 384, -60);
+  ASSERT_EQ(GetMask(cdc8), 0xFFFFFFFFFFFFFFFF);
+}
+
+TEST_F(TestCDC, WriteSingleColumnParquetFile) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  std::vector<int32_t> numbers = {1, 2, 3, 4, 5};
+  std::vector<uint8_t> valid_bits = {1, 0, 1, 0, 1};
+  EXPECT_THROW(
+      int_column_writer.WriteBatch(numbers.size(), nullptr, nullptr, numbers.data()),
+      ParquetException);
+  EXPECT_THROW(int_column_writer.WriteBatchSpaced(numbers.size(), nullptr, nullptr,
+                                                  valid_bits.data(), 0, numbers.data()),
+               ParquetException);
+}
+
+TEST_F(TestCDC, LastChunkDoesntTriggerAddDataPage) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()
+                   ->content_defined_chunking_options(kMinChunkSize, kMaxChunkSize, 0)
+                   ->disable_dictionary()
+                   ->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  ASSERT_OK_AND_ASSIGN(auto array, ::arrow::gen::Step()->Generate(8000));
+  auto arrow_props = default_arrow_writer_properties();
+  auto arrow_ctx = ArrowWriteContext(default_memory_pool(), arrow_props.get());
+
+  // Calling WriteArrow twice, we expect that the first call doesn't add a new data page
+  // at the end allowing subsequent calls to append to the same page
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+
+  int_column_writer.Close();
+  writer->Close();
+  ASSERT_OK_AND_ASSIGN(auto buffer, sink->Finish());
+
+  auto info = GetColumnParquetInfo(buffer);
+  ASSERT_EQ(info.size(), 1);
+
+  // AssertAllBetween allow the last chunk size to be smaller than the min_chunk_size
+  AssertAllBetween(info[0].page_sizes, kMinChunkSize, kMaxChunkSize);
+  AssertAllBetween(info[0].page_lengths, 3000, 5000);
+}
+
+TEST_F(TestCDC, ChunkSizeParameterValidation) {
+  // Test that constructor validates min/max chunk size parameters
+  auto li = LevelInfo();
+
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 256 * 1024, 1024 * 1024));
+
+  // with norm_factor=0 the difference between min and max chunk size must be
+  // at least 16
+  ASSERT_THROW(ContentDefinedChunker(li, 0, -1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1024, 512), ParquetException);
+
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16));
+  ASSERT_THROW(ContentDefinedChunker(li, -16, -16), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 16, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 32, 32), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 32, 48));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1024 * 1024, 2 * 1024 * 1024));
+  ASSERT_NO_THROW(
+      ContentDefinedChunker(li, 1024 * 1024 * 1024L, 2LL * 1024 * 1024 * 1024L));
+
+  // with norm_factor=1 the difference between min and max chunk size must be
+  // at least 64
+  ASSERT_THROW(ContentDefinedChunker(li, 1, -1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 32, 1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1, 33, 1));
+
+  // with norm_factor=2 the difference between min and max chunk size must be
+  // at least 128
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 63, 2), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 64, 2));
+
+  // with norm_factor=-1 the difference between min and max chunk size must be
+  // at least 8
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 7, -1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 8, -1));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16, -1));
+
+  // test the norm_factor extremes
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0, -68), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 1, -67));
+  ASSERT_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 59),
+               ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 58));
+}
+
+class TestCDCSingleRowGroup : public ::testing::TestWithParam<CaseConfig> {
+ protected:
+  // Column random table parts for testing
+  std::shared_ptr<Field> field_;
+  std::shared_ptr<Table> part1_, part2_, part3_, part4_, part5_, part6_, part7_;
+
+  void SetUp() override {
+    const auto& param = GetParam();
+    auto field_ = ::arrow::field("f0", param.dtype, param.is_nullable);
+    auto schema = ::arrow::schema({field_});
+
+    auto part_length = kPartSize / param.bytes_per_record;
+    auto edit_length = kEditSize / param.bytes_per_record;
+    ASSERT_OK_AND_ASSIGN(part1_, GenerateTable(schema, part_length, 0));
+    ASSERT_OK_AND_ASSIGN(part2_, GenerateTable(schema, edit_length, 1));
+    ASSERT_OK_AND_ASSIGN(part3_, GenerateTable(schema, part_length, part_length));
+    ASSERT_OK_AND_ASSIGN(part4_, GenerateTable(schema, edit_length, 2));
+    ASSERT_OK_AND_ASSIGN(part5_, GenerateTable(schema, part_length, 2 * part_length));
+    ASSERT_OK_AND_ASSIGN(part6_, GenerateTable(schema, edit_length, 3));
+    ASSERT_OK_AND_ASSIGN(part7_, GenerateTable(schema, edit_length, 4));
+  }
+};
+
+TEST_P(TestCDCSingleRowGroup, DeleteOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/0,
+                                /*exact_number_of_smaller_diffs=*/1, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, DeleteTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part3_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/0,
+                                /*exact_number_of_smaller_diffs=*/2, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, UpdateOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part4_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*max_number_of_equal_diffs=*/1);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, UpdateTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified,
+                       ConcatAndCombine({part1_, part6_, part3_, part7_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*max_number_of_equal_diffs=*/2);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, InsertOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/1,
+                                /*exact_number_of_smaller_diffs=*/0, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, InsertTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part3_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/2,
+                                /*exact_number_of_smaller_diffs=*/0, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, Append) {

Review Comment:
   Added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025090725


##########
cpp/src/parquet/chunker_internal_codegen.py:
##########
@@ -0,0 +1,121 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Produce the given number gearhash tables for rolling hash calculations.
+
+Each table consists of 256 64-bit integer values and by default 8 tables are
+produced. The tables are written to a header file that can be included in the
+C++ code.
+
+The generated numbers are deterministic "random" numbers created by MD5 hashing
+a fixed seed and the table index. This ensures that the tables are the same
+across different runs and platforms. The function of generating the numbers is
+less important as long as they have sufficiently uniform distribution.
+
+Reference implementations:
+- https://github.com/Borelset/destor/blob/master/src/chunking/fascdc_chunking.c
+- https://github.com/nlfiedler/fastcdc-rs/blob/master/examples/table64.rs
+
+Usage:
+    python chunker_internal_codegen.py [ntables]
+
+    ntables: Number of gearhash tables to generate (default 8), the
+             the C++ implementation expects 8 tables so this should not be
+             changed unless the C++ code is also updated.
+
+    The generated header file is written to ./chunker_internal_generated.h
+"""
+
+import hashlib
+import pathlib
+import sys
+from io import StringIO
+
+
+template = """\
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+#include <cstdint>
+
+namespace parquet::internal {{
+
+constexpr uint64_t kGearhashTable[8][256] = {{

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027904963


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical

Review Comment:
   During the development it wasn't always obvious :)
   Removed the redundant checks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2876952223

   @github-actions crossbow submit preview-docs


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2083092149


##########
cpp/src/parquet/properties.h:
##########
@@ -245,6 +245,34 @@ class PARQUET_EXPORT ColumnProperties {
   bool page_index_enabled_;
 };
 
+// EXPERIMENTAL: Options for content-defined chunking.
+struct PARQUET_EXPORT CdcOptions {
+  /// Minimum chunk size in bytes, default 256 KiB
+  /// The rolling hash will not be updated until this size is reached for each chunk.
+  /// Note that all data sent through the hash function is counted towards the chunk
+  /// size, including definition and repetition levels if present.
+  int64_t min_chunk_size;

Review Comment:
   Good idea! Updating.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2083099437


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,408 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,

Review Comment:
   Thanks! I renamed norm_factor to norm_level everywhere. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086408290


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1689 @@
+// Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   And `parquet-chunker-test` took [5 minutes](https://github.com/ursacomputing/crossbow/actions/runs/14991686603/job/42116307723#step:6:5280) on Valgrind.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086563061


##########
docs/source/python/parquet.rst:
##########
@@ -782,3 +782,63 @@ file decryption properties) is optional and it includes the following options:
 
 * ``cache_lifetime``, the lifetime of cached entities (key encryption keys, local
   wrapping keys, KMS client objects) represented as a ``datetime.timedelta``.
+
+
+Content-Defined Chunking
+------------------------
+
+.. note::
+   This feature is experimental and may change in future releases.
+
+PyArrow introduces an experimental feature for optimizing Parquet files for content
+addressable storage (CAS) systems using content-defined chunking (CDC). This feature
+enables efficient deduplication of data across files, improving network transfers and
+storage efficiency.
+
+When enabled, data pages are written according to content-defined chunk boundaries,
+determined by a rolling hash algorithm that identifies chunk boundaries based on the
+actual content of the data. When data in a column is modified (e.g., inserted, deleted,
+or updated), this approach minimizes the number of changed data pages.
+
+The feature can be enabled by setting the ``use_content_defined_chunking`` parameter in
+the Parquet writer. It accepts either a boolean or a dictionary for configuration:
+
+- ``True``: Uses the default configuration with:
+   - Minimum chunk size: 256 KiB
+   - Maximum chunk size: 1024 KiB
+   - Normalization level: 0
+
+- ``dict``: Allows customization of the chunking parameters:
+   - ``min_chunk_size``: Minimum chunk size in bytes (default: 256 KiB).
+   - ``max_chunk_size``: Maximum chunk size in bytes (default: 1024 KiB).
+   - ``norm_level``: Normalization level to adjust chunk size distribution (default: 0).
+
+Note that the chunk size is calculated on the logical values before applying any encoding
+or compression. The actual size of the data pages may vary based on the encoding and
+compression used.
+
+.. note::
+   Ensure that Parquet write options remain consistent across writes and files.
+   Using different write options (like compression, encoding, or row group size)
+   for different files may prevent proper deduplication and lead to suboptimal
+   storage efficiency.

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086137223


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,412 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+namespace {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_level parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_level Normalization level (default 0)
+// @return The mask used to compare against the rolling hash
+uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size, int norm_level) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be non-negative");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes; also divide by the number of gearhash tables to have a
+  // a more gaussian-like distribution
+  int64_t target_size = (avg_chunk_size - min_chunk_size) / kNumGearhashTables;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // a user defined `norm_level` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_level we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_level is 0 by
+  // default
+  int effective_bits = mask_bits - norm_level;
+
+  if (effective_bits < 1 || effective_bits > 63) {
+    throw ParquetException(
+        "The number of bits in the CDC mask must be between 1 and 63, got " +
+        std::to_string(effective_bits));
+  } else {
+    // create the mask by setting the top bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+}  // namespace
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int norm_level)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_level)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the

Review Comment:
   Agree, then how about `CDC => json-encoded-params`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086135451


##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,144 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <vector>
+
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  // The start offset of this chunk inside the given levels
+  int64_t level_offset;
+  // The start offset of this chunk inside the given values array
+  int64_t value_offset;
+  // The length of the chunk in levels
+  int64_t levels_to_write;
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which is contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...

Review Comment:
   I think that is a made-up example, not actual data that you can reproduce with specific CDC settings :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2083096080


##########
cpp/src/parquet/properties.h:
##########
@@ -245,6 +245,34 @@ class PARQUET_EXPORT ColumnProperties {
   bool page_index_enabled_;
 };
 
+// EXPERIMENTAL: Options for content-defined chunking.
+struct PARQUET_EXPORT CdcOptions {
+  /// Minimum chunk size in bytes, default 256 KiB
+  /// The rolling hash will not be updated until this size is reached for each chunk.
+  /// Note that all data sent through the hash function is counted towards the chunk
+  /// size, including definition and repetition levels if present.
+  int64_t min_chunk_size;

Review Comment:
   Updated to set the default here and initialize kDefaultCdcOptions using those.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025064635


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int8_t norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes
+  int64_t target_size = avg_chunk_size - min_chunk_size;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // 3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  if (norm_factor < -3 || norm_factor > 3) {
+    ARROW_LOG(WARNING) << "norm_factor=" << norm_factor
+                       << " is outside the recommended range [-3, 3]";
+  }

Review Comment:
   Removed, explicitly mentioned in properties.h that it means bit adjustment to the gear hash mask.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025142266


##########
cpp/src/parquet/properties.h:
##########
@@ -730,6 +791,9 @@ class PARQUET_EXPORT WriterProperties {
     std::unordered_map<std::string, bool> dictionary_enabled_;
     std::unordered_map<std::string, bool> statistics_enabled_;
     std::unordered_map<std::string, bool> page_index_enabled_;
+
+    bool content_defined_chunking_enabled_;
+    CDCOptions content_defined_chunking_options_;

Review Comment:
   The default values for the builder fields are set from the `Builder` constructor so I am just following the convention. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["wgtmac (via GitHub)" <gi...@apache.org>]

wgtmac commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1978477938


##########
cpp/src/parquet/column_chunker.h:
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+
+namespace internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage systems (CAS)
+/// which split the bytes stream into content defined blobs. The CAS system will calculate
+/// a unique identifier for each blob, then store the blob in a key-value store. If the
+/// same blob is encountered again, the system can refer to the hash instead of physically
+/// storing the blob again. In the example above, the CAS system would phiysically store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks///
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC paper: "FastCDC: a Fast and Efficient Content-Defined Chunking Approach for
+/// Data Deduplication"
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+class ContentDefinedChunker {

Review Comment:
   IIUC, this file can be renamed to `column_chunk_internal.h` to avoid being installed accidentally.



##########
cpp/src/parquet/column_writer.h:
##########
@@ -23,6 +23,7 @@
 
 #include "arrow/type_fwd.h"
 #include "arrow/util/compression.h"
+#include "parquet/column_chunker.h"

Review Comment:
   This seems unnecessary 



##########
cpp/src/parquet/column_chunker.h:
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+
+namespace internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage systems (CAS)
+/// which split the bytes stream into content defined blobs. The CAS system will calculate
+/// a unique identifier for each blob, then store the blob in a key-value store. If the
+/// same blob is encountered again, the system can refer to the hash instead of physically
+/// storing the blob again. In the example above, the CAS system would phiysically store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.

Review Comment:
   ```suggestion
   /// Then the parquet file is being uploaded to a content addressable storage system (CAS)
   /// which splits the bytes stream into content defined blobs. The CAS system will calculate
   /// an unique identifier for each blob, then store the blob in a key-value store. If the
   /// same blob is encountered again, the system can refer to the hash instead of physically
   /// storing the blob again. In the example above, the CAS system would physically store
   /// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
   /// files.
   ```



##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1332,13 +1337,38 @@ class TypedColumnWriterImpl : public ColumnWriterImpl, public TypedColumnWriter<
       bits_buffer_->ZeroPadding();
     }
 
-    if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
-      return WriteArrowDictionary(def_levels, rep_levels, num_levels, leaf_array, ctx,
-                                  maybe_parent_nulls);
+    if (properties_->cdc_enabled()) {
+      ARROW_ASSIGN_OR_RAISE(auto boundaries,
+                            content_defined_chunker_.GetBoundaries(
+                                def_levels, rep_levels, num_levels, leaf_array));
+      for (auto chunk : boundaries) {
+        auto chunk_array = leaf_array.Slice(chunk.value_offset);
+        auto chunk_def_levels = AddIfNotNull(def_levels, chunk.level_offset);
+        auto chunk_rep_levels = AddIfNotNull(rep_levels, chunk.level_offset);
+        if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {

Review Comment:
   Do we consider same value tuples to be equal even when they are not encoded in the same way? For example, one is dictionary-encoded and the other is not but their values are the same. Same question for other encodings (e.g. plain vs byte_stream_split for double type).



##########
cpp/src/parquet/properties.h:
##########
@@ -275,10 +282,33 @@ class PARQUET_EXPORT WriterProperties {
           page_checksum_enabled_(properties.page_checksum_enabled()),
           size_statistics_level_(properties.size_statistics_level()),
           sorting_columns_(properties.sorting_columns()),
-          default_column_properties_(properties.default_column_properties()) {}
+          default_column_properties_(properties.default_column_properties()),
+          cdc_enabled_(properties.cdc_enabled()),
+          cdc_size_range_(properties.cdc_size_range()),
+          cdc_norm_factor_(properties.cdc_norm_factor()) {}
 
     virtual ~Builder() {}
 
+    Builder* enable_cdc() {

Review Comment:
   What about using its full name instead of `cdc`? It may not be obvious to people who are not familiar with it and be mixed with another norm `change data capture`.



##########
cpp/src/parquet/column_chunker.h:
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+
+namespace internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage systems (CAS)
+/// which split the bytes stream into content defined blobs. The CAS system will calculate
+/// a unique identifier for each blob, then store the blob in a key-value store. If the
+/// same blob is encountered again, the system can refer to the hash instead of physically
+/// storing the blob again. In the example above, the CAS system would phiysically store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks///
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC paper: "FastCDC: a Fast and Efficient Content-Defined Chunking Approach for
+/// Data Deduplication"
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+class ContentDefinedChunker {
+ public:
+  /// Create a new ContentDefinedChunker instance
+  ///
+  /// @param level_info Information about definition and repetition levels
+  /// @param size_range Min/max chunk size as pair<min_size, max_size>, the chunker will
+  ///                   attempt to uniformly distribute the chunks between these extremes.
+  /// @param norm_factor Normalization factor to center the chunk size around the average
+  ///                    size more aggressively. By increasing the normalization factor,
+  ///                    probability of finding a chunk boundary increases.
+  ContentDefinedChunker(const LevelInfo& level_info,
+                        std::pair<uint64_t, uint64_t> size_range,
+                        uint8_t norm_factor = 0);
+
+  /// Get the chunk boundaries for the given column data
+  ///
+  /// @param def_levels Definition levels
+  /// @param rep_levels Repetition levels
+  /// @param num_levels Number of levels
+  /// @param values Column values as an Arrow array
+  /// @return Vector of Chunk objects representing the chunk boundaries
+  const ::arrow::Result<std::vector<Chunk>> GetBoundaries(const int16_t* def_levels,

Review Comment:
   When will the `::arrow::Result` hold an error? Usually in the parquet module we use throw instead of status.



##########
cpp/src/parquet/column_chunker.h:
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+
+namespace internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage systems (CAS)
+/// which split the bytes stream into content defined blobs. The CAS system will calculate
+/// a unique identifier for each blob, then store the blob in a key-value store. If the
+/// same blob is encountered again, the system can refer to the hash instead of physically
+/// storing the blob again. In the example above, the CAS system would phiysically store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks///

Review Comment:
   ```suggestion
   /// Chunk objects that represent the boundaries of the chunks.
   ```



##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1332,13 +1337,38 @@ class TypedColumnWriterImpl : public ColumnWriterImpl, public TypedColumnWriter<
       bits_buffer_->ZeroPadding();
     }
 
-    if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
-      return WriteArrowDictionary(def_levels, rep_levels, num_levels, leaf_array, ctx,
-                                  maybe_parent_nulls);
+    if (properties_->cdc_enabled()) {
+      ARROW_ASSIGN_OR_RAISE(auto boundaries,
+                            content_defined_chunker_.GetBoundaries(

Review Comment:
   There are some cases where the parquet writer will further split the input Arrow array into smaller pieces which may affect the precision of the CDC logic here:
   
   - Split the input for max_row_group_size: https://github.com/apache/arrow/blob/main/cpp/src/parquet/arrow/writer.cc#L458-L470
   - If data page v2 or page index is enabled, page boundary must be a record boundary (i.e. rep_level = 0), this prohibits page cut at certain values: https://github.com/apache/arrow/blob/main/cpp/src/parquet/column_writer.cc#L1142-L1188



##########
cpp/src/parquet/column_chunker.h:
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+
+namespace internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage systems (CAS)
+/// which split the bytes stream into content defined blobs. The CAS system will calculate
+/// a unique identifier for each blob, then store the blob in a key-value store. If the
+/// same blob is encountered again, the system can refer to the hash instead of physically
+/// storing the blob again. In the example above, the CAS system would phiysically store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks///
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC paper: "FastCDC: a Fast and Efficient Content-Defined Chunking Approach for
+/// Data Deduplication"
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+class ContentDefinedChunker {
+ public:
+  /// Create a new ContentDefinedChunker instance
+  ///
+  /// @param level_info Information about definition and repetition levels
+  /// @param size_range Min/max chunk size as pair<min_size, max_size>, the chunker will
+  ///                   attempt to uniformly distribute the chunks between these extremes.
+  /// @param norm_factor Normalization factor to center the chunk size around the average
+  ///                    size more aggressively. By increasing the normalization factor,
+  ///                    probability of finding a chunk boundary increases.
+  ContentDefinedChunker(const LevelInfo& level_info,
+                        std::pair<uint64_t, uint64_t> size_range,
+                        uint8_t norm_factor = 0);
+
+  /// Get the chunk boundaries for the given column data
+  ///
+  /// @param def_levels Definition levels
+  /// @param rep_levels Repetition levels
+  /// @param num_levels Number of levels
+  /// @param values Column values as an Arrow array
+  /// @return Vector of Chunk objects representing the chunk boundaries
+  const ::arrow::Result<std::vector<Chunk>> GetBoundaries(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const ::arrow::Array& values);

Review Comment:
   For `::arrow::Array` we assume that CDC is not supported in the methods provided by TypedColumnWriter: https://github.com/apache/arrow/blob/main/cpp/src/parquet/column_writer.h#L196 ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1968426044


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {

Review Comment:
   We have 8 independent gear hash tables in order to normalize chunk sizes, see more details [in the comments here](https://github.com/apache/arrow/pull/45360/files#diff-b0e0dc9efa0271723fc8f200c7a54465dd8)
   
   It helps to achieve chunk size distributions like the following:
   ![newplot-8](https://github.com/user-attachments/assets/9b3224df-e495-4d05-8307-4ca930096da1)
   61655788434ee2884d9bac72e83c2R622).
   
   The spikes on the left are because of the dictionary encoding and the right because of the max chunk size limitation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2679654575

   @pitrou @emkornfield @mapleFU @wgtmac could you please take a look? I am trying to finalize the PR and I could use some feedback from you. 
   
   The testing itself is rather challenging since the chunks depend on the actual data. Currently I am generating random arrow tables because that has complete type support whereas the generators under `arrow/testing/generator.h` are rather incomplete. 
   
   Also the test cases are rather slow, I tried to reduce the generated data samples but the heuristics used in the assertions are getting more pronounced and harder to reliably work with.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1974001697


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+bool ContentDefinedChunker::Check() {

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086657214


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1689 @@
+// Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   I have a test suite configuration mostly for completeness:
   
   ```cpp
   INSTANTIATE_TEST_SUITE_P(
       FixedSizedTypes, TestCDCSingleRowGroup,
       testing::Values(
           // Boolean
           CaseConfig{::arrow::boolean(), false},
           // Numeric
           CaseConfig{::arrow::uint8(), false}, CaseConfig{::arrow::uint16(), false},
           CaseConfig{::arrow::uint32(), false}, CaseConfig{::arrow::uint64(), true},
           CaseConfig{::arrow::int8(), false}, CaseConfig{::arrow::int16(), false},
           CaseConfig{::arrow::int32(), false}, CaseConfig{::arrow::int64(), true},
           CaseConfig{::arrow::float16(), false}, CaseConfig{::arrow::float32(), false},
           CaseConfig{::arrow::float64(), true},
           CaseConfig{::arrow::decimal128(18, 6), false},
           CaseConfig{::arrow::decimal256(40, 6), false},
           // Binary-like
           CaseConfig{::arrow::utf8(), false}, CaseConfig{::arrow::binary(), true},
           CaseConfig{::arrow::fixed_size_binary(16), true},
           // Temporal
           CaseConfig{::arrow::date32(), false},
           CaseConfig{::arrow::time32(::arrow::TimeUnit::MILLI), true},
           CaseConfig{::arrow::time64(::arrow::TimeUnit::NANO), false},
           CaseConfig{::arrow::timestamp(::arrow::TimeUnit::NANO), true},
           CaseConfig{::arrow::duration(::arrow::TimeUnit::NANO), false},
           // Nested types
           CaseConfig{::arrow::list(::arrow::int32()), false},
           CaseConfig{::arrow::list(::arrow::int32()), true},
           CaseConfig{::arrow::list(::arrow::utf8()), true},
           CaseConfig{::arrow::struct_({::arrow::field("f0", ::arrow::int32())}), false},
           CaseConfig{::arrow::struct_({::arrow::field("f0", ::arrow::float64())}), true},
           CaseConfig{
               ::arrow::list(::arrow::struct_({::arrow::field("f0", ::arrow::int32())})),
               false},
           // Extension type
           CaseConfig{::arrow::extension::json(), true},
           // Use ParquetDataPageVersion::V2
           CaseConfig{::arrow::large_binary(), false, ParquetDataPageVersion::V2},
           CaseConfig{::arrow::list(::arrow::utf8()), true, ParquetDataPageVersion::V2}));
   ```
   
   I can split this into two sets of cases, by default a small set would be exercised but could be explicitly enabled by a gtest flag. Opinions?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2017652290


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size

Review Comment:
   Let's defer this to a follow-up improvement. The `norm_factor` parameter should rarely be used.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2017651577


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+bool ContentDefinedChunker::Check() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last Check() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {

Review Comment:
   You are right, max chunk size is checked after updating the rolling hash, it works similarly to `max_page_size` which is also being checked after appending the value. 
   
   If we were checking the max chunk size before updating the rolling hash, in case of a large record we may encounter the case that single record doesn't fit in max_chunk_size so it would endlessly try to close the chunk and fit the record to a new one where it doesn't fit again. 
   
   This could be properly implemented of course but would require more complicated branches, also enforcing the record boundary would be harder. I think this is a trade-off we should keep, at least for now. Maybe in a follow-up I can properly enforce the max chunk size with more tests and some benchmarks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2760723174

   Revision: a378de195a3d35a7b8d4e07d8d3342696667f927
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-f7a963d6fd](https://github.com/ursacomputing/crossbow/branches/all?query=actions-f7a963d6fd)
   
   |Task|Status|
   |----|------|
   |wheel-macos-monterey-cp310-cp310-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-macos-monterey-cp310-cp310-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057506/job/39575298524)|
   |wheel-macos-monterey-cp310-cp310-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-macos-monterey-cp310-cp310-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126058318/job/39575303112)|
   |wheel-macos-monterey-cp311-cp311-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-macos-monterey-cp311-cp311-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126058219/job/39575302441)|
   |wheel-macos-monterey-cp311-cp311-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-macos-monterey-cp311-cp311-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126058145/job/39575302314)|
   |wheel-macos-monterey-cp312-cp312-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-macos-monterey-cp312-cp312-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057127/job/39575296313)|
   |wheel-macos-monterey-cp312-cp312-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-macos-monterey-cp312-cp312-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057616/job/39575299503)|
   |wheel-macos-monterey-cp313-cp313-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-macos-monterey-cp313-cp313-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057621/job/39575299498)|
   |wheel-macos-monterey-cp313-cp313-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-macos-monterey-cp313-cp313-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057462/job/39575297440)|
   |wheel-macos-monterey-cp313-cp313t-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-macos-monterey-cp313-cp313t-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057950/job/39575301397)|
   |wheel-macos-monterey-cp313-cp313t-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-macos-monterey-cp313-cp313t-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057881/job/39575301214)|
   |wheel-macos-monterey-cp39-cp39-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-macos-monterey-cp39-cp39-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057339/job/39575297529)|
   |wheel-macos-monterey-cp39-cp39-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-macos-monterey-cp39-cp39-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057430/job/39575297412)|
   |wheel-manylinux-2-28-cp310-cp310-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2-28-cp310-cp310-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057354/job/39575298057)|
   |wheel-manylinux-2-28-cp310-cp310-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2-28-cp310-cp310-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126056998/job/39575295396)|
   |wheel-manylinux-2-28-cp311-cp311-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2-28-cp311-cp311-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057186/job/39575296856)|
   |wheel-manylinux-2-28-cp311-cp311-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2-28-cp311-cp311-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057612/job/39575299510)|
   |wheel-manylinux-2-28-cp312-cp312-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2-28-cp312-cp312-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057001/job/39575295304)|
   |wheel-manylinux-2-28-cp312-cp312-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2-28-cp312-cp312-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126058113/job/39575302146)|
   |wheel-manylinux-2-28-cp313-cp313-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2-28-cp313-cp313-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057374/job/39575297794)|
   |wheel-manylinux-2-28-cp313-cp313-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2-28-cp313-cp313-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126058048/job/39575301857)|
   |wheel-manylinux-2-28-cp313-cp313t-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2-28-cp313-cp313t-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126058300/job/39575302940)|
   |wheel-manylinux-2-28-cp313-cp313t-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2-28-cp313-cp313t-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057497/job/39575298369)|
   |wheel-manylinux-2-28-cp39-cp39-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2-28-cp39-cp39-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057107/job/39575296275)|
   |wheel-manylinux-2-28-cp39-cp39-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2-28-cp39-cp39-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126056855/job/39575295044)|
   |wheel-manylinux-2014-cp310-cp310-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2014-cp310-cp310-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126058373/job/39575303516)|
   |wheel-manylinux-2014-cp310-cp310-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2014-cp310-cp310-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057783/job/39575300560)|
   |wheel-manylinux-2014-cp311-cp311-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2014-cp311-cp311-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057680/job/39575299974)|
   |wheel-manylinux-2014-cp311-cp311-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2014-cp311-cp311-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057134/job/39575296577)|
   |wheel-manylinux-2014-cp312-cp312-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2014-cp312-cp312-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057244/job/39575297133)|
   |wheel-manylinux-2014-cp312-cp312-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2014-cp312-cp312-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126058027/job/39575301714)|
   |wheel-manylinux-2014-cp313-cp313-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2014-cp313-cp313-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057420/job/39575297598)|
   |wheel-manylinux-2014-cp313-cp313-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2014-cp313-cp313-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126056854/job/39575295257)|
   |wheel-manylinux-2014-cp313-cp313t-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2014-cp313-cp313t-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126058250/job/39575302596)|
   |wheel-manylinux-2014-cp313-cp313t-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2014-cp313-cp313t-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126056897/job/39575295237)|
   |wheel-manylinux-2014-cp39-cp39-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2014-cp39-cp39-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057513/job/39575299123)|
   |wheel-manylinux-2014-cp39-cp39-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-manylinux-2014-cp39-cp39-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126058181/job/39575302427)|
   |wheel-musllinux-1-2-cp310-cp310-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-musllinux-1-2-cp310-cp310-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057203/job/39575297173)|
   |wheel-musllinux-1-2-cp310-cp310-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-musllinux-1-2-cp310-cp310-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057980/job/39575301536)|
   |wheel-musllinux-1-2-cp311-cp311-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-musllinux-1-2-cp311-cp311-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057576/job/39575299319)|
   |wheel-musllinux-1-2-cp311-cp311-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-musllinux-1-2-cp311-cp311-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057810/job/39575300754)|
   |wheel-musllinux-1-2-cp312-cp312-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-musllinux-1-2-cp312-cp312-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057047/job/39575295975)|
   |wheel-musllinux-1-2-cp312-cp312-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-musllinux-1-2-cp312-cp312-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126058338/job/39575303323)|
   |wheel-musllinux-1-2-cp313-cp313-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-musllinux-1-2-cp313-cp313-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057084/job/39575295982)|
   |wheel-musllinux-1-2-cp313-cp313-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-musllinux-1-2-cp313-cp313-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126058068/job/39575302173)|
   |wheel-musllinux-1-2-cp313-cp313t-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-musllinux-1-2-cp313-cp313t-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126056838/job/39575295024)|
   |wheel-musllinux-1-2-cp313-cp313t-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-musllinux-1-2-cp313-cp313t-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057496/job/39575298426)|
   |wheel-musllinux-1-2-cp39-cp39-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-musllinux-1-2-cp39-cp39-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057510/job/39575298703)|
   |wheel-musllinux-1-2-cp39-cp39-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-musllinux-1-2-cp39-cp39-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/14126058184/job/39575302461)|
   |wheel-windows-cp310-cp310-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-windows-cp310-cp310-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126058290/job/39575302907)|
   |wheel-windows-cp311-cp311-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-windows-cp311-cp311-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057499/job/39575298221)|
   |wheel-windows-cp312-cp312-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-windows-cp312-cp312-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126058285/job/39575302697)|
   |wheel-windows-cp313-cp313-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-windows-cp313-cp313-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057799/job/39575300711)|
   |wheel-windows-cp313-cp313t-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-windows-cp313-cp313t-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126058362/job/39575303530)|
   |wheel-windows-cp39-cp39-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-f7a963d6fd-github-wheel-windows-cp39-cp39-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14126057424/job/39575298065)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2083089272


##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -395,6 +396,8 @@ add_parquet_test(writer-test
                  file_serialize_test.cc
                  stream_writer_test.cc)
 
+add_parquet_test(chunker-test SOURCES chunker_internal_test.cc)

Review Comment:
   The CDC tests are much slower than the writer tests, so I thought it is better to separate them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2083097850


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2868746114

   > Sorry for the delay! I haven't finished the review yet. Will continue this weekend.
   
   No worries, I really appreciate that you take the time to review! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["wgtmac (via GitHub)" <gi...@apache.org>]

wgtmac commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2081229168


##########
cpp/src/parquet/properties.h:
##########
@@ -245,6 +245,34 @@ class PARQUET_EXPORT ColumnProperties {
   bool page_index_enabled_;
 };
 
+// EXPERIMENTAL: Options for content-defined chunking.
+struct PARQUET_EXPORT CdcOptions {
+  /// Minimum chunk size in bytes, default 256 KiB
+  /// The rolling hash will not be updated until this size is reached for each chunk.
+  /// Note that all data sent through the hash function is counted towards the chunk
+  /// size, including definition and repetition levels if present.
+  int64_t min_chunk_size;

Review Comment:
   Should we set the default value to avoid unintentionally uninitialized value?



##########
cpp/src/parquet/chunker_internal_codegen.py:
##########
@@ -0,0 +1,124 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Produce the given number gearhash tables for rolling hash calculations.
+
+Each table consists of 256 64-bit integer values and by default 8 tables are
+produced. The tables are written to a header file that can be included in the
+C++ code.
+
+The generated numbers are deterministic "random" numbers created by MD5 hashing
+a fixed seed and the table index. This ensures that the tables are the same
+across different runs and platforms. The function of generating the numbers is
+less important as long as they have sufficiently uniform distribution.
+
+Reference implementations:
+- https://github.com/Borelset/destor/blob/master/src/chunking/fascdc_chunking.c
+- https://github.com/nlfiedler/fastcdc-rs/blob/master/examples/table64.rs
+
+Usage:
+    python chunker_internal_codegen.py [ntables]
+
+    ntables: Number of gearhash tables to generate (default 8), the
+             the C++ implementation expects 8 tables so this should not be
+             changed unless the C++ code is also updated.
+
+    The generated header file is written to ./chunker_internal_generated.h
+"""
+
+import hashlib
+import pathlib
+import sys
+from io import StringIO
+
+
+template = """\
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+#include <cstdint>

Review Comment:
   ```suggestion
   #pragma once
   
   #include <cstdint>
   ```
   Leave a blank line in between?
   



##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -395,6 +396,8 @@ add_parquet_test(writer-test
                  file_serialize_test.cc
                  stream_writer_test.cc)
 
+add_parquet_test(chunker-test SOURCES chunker_internal_test.cc)

Review Comment:
   Why not reusing `writer-test` above?



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,408 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,

Review Comment:
   Level sounds better since I also assumed that it will multiply some values.



##########
cpp/src/parquet/properties.h:
##########
@@ -245,6 +245,34 @@ class PARQUET_EXPORT ColumnProperties {
   bool page_index_enabled_;
 };
 
+// EXPERIMENTAL: Options for content-defined chunking.
+struct PARQUET_EXPORT CdcOptions {
+  /// Minimum chunk size in bytes, default 256 KiB
+  /// The rolling hash will not be updated until this size is reached for each chunk.
+  /// Note that all data sent through the hash function is counted towards the chunk
+  /// size, including definition and repetition levels if present.
+  int64_t min_chunk_size;
+  /// Maximum chunk size in bytes, default is 1024 KiB
+  /// The chunker will create a new chunk whenever the chunk size exceeds this value.
+  /// Note that the parquet writer has a related `pagesize` property that controls
+  /// the maximum size of a parquet data page after encoding. While setting
+  /// `pagesize` to a smaller value than `max_chunk_size` doesn't affect the
+  /// chunking effectiveness, it results in more small parquet data pages.
+  int64_t max_chunk_size;
+  /// Number of bit adjustement to the gearhash mask in order to
+  /// center the chunk size around the average size more aggressively, default 0
+  /// Increasing the normalization factor increases the probability of finding a chunk,
+  /// improving the deduplication ratio, but also increasing the number of small chunks
+  /// resulting in many small parquet data pages. The default value provides a good
+  /// balance between deduplication ratio and fragmentation. Use norm_factor=1 or
+  /// norm_factor=2 to reach a higher deduplication ratio at the expense of
+  /// fragmentation. Negative values can also be used to reduce the probability of
+  /// finding a chunk, resulting in larger chunks and fewer data pages.
+  int norm_factor = 0;

Review Comment:
   Is there any bound to norm_factor? Should we document it?



##########
cpp/src/parquet/properties.h:
##########
@@ -275,10 +305,38 @@ class PARQUET_EXPORT WriterProperties {
           page_checksum_enabled_(properties.page_checksum_enabled()),
           size_statistics_level_(properties.size_statistics_level()),
           sorting_columns_(properties.sorting_columns()),
-          default_column_properties_(properties.default_column_properties()) {}
+          default_column_properties_(properties.default_column_properties()),
+          content_defined_chunking_enabled_(
+              properties.content_defined_chunking_enabled()),
+          content_defined_chunking_options_(
+              properties.content_defined_chunking_options()) {}
 
     virtual ~Builder() {}
 
+    /// \brief EXPERIMENTAL: Use content-defined page chunking for all columns.
+    ///
+    /// Optimize parquet files for content addressable storage (CAS) systems by writing
+    /// data pages according to content-defined chunk boundaries. This allows for more
+    /// efficient deduplication of data across files, hence more efficient network
+    /// transfers and storage. The chunking is based on a rolling hash algorithm that
+    /// identifies chunk boundaries based on the actual content of the data.
+    Builder* enable_content_defined_chunking() {
+      content_defined_chunking_enabled_ = true;
+      return this;
+    }
+
+    /// \brief EXPERIMENTAL: Disable content-defined page chunking for all columns.
+    Builder* disable_content_defined_chunking() {
+      content_defined_chunking_enabled_ = false;
+      return this;
+    }
+
+    /// \brief EXPERIMENTAL: Specify content-defined chunking options, see CdcOptions.
+    Builder* content_defined_chunking_options(const CdcOptions options) {

Review Comment:
   ```suggestion
       Builder* content_defined_chunking_options(CdcOptions options) {
   ```
   
   Or `const CdcOptions&` ?



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes; also divide by the number of gearhash tables to have a
+  // a more gaussian-like distribution
+  int64_t target_size = (avg_chunk_size - min_chunk_size) / kNumGearhashTables;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  int effective_bits = mask_bits - norm_factor;
+
+  if (effective_bits < 1 || effective_bits > 63) {
+    throw ParquetException(
+        "The number of bits in the CDC mask must be between 1 and 63, got " +
+        std::to_string(effective_bits));
+  } else {
+    // create the mask by setting the top bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int norm_factor)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_factor)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(const bool value) {

Review Comment:
   ```suggestion
     void Roll(bool value) {
   ```



##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1337,13 +1368,47 @@ class TypedColumnWriterImpl : public ColumnWriterImpl,
       bits_buffer_->ZeroPadding();
     }
 
-    if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
-      return WriteArrowDictionary(def_levels, rep_levels, num_levels, leaf_array, ctx,
-                                  maybe_parent_nulls);
+    if (properties_->content_defined_chunking_enabled()) {
+      DCHECK(content_defined_chunker_.has_value());
+      auto chunks = content_defined_chunker_->GetChunks(def_levels, rep_levels,
+                                                        num_levels, leaf_array);
+      for (size_t i = 0; i < chunks.size(); i++) {
+        auto chunk = chunks[i];
+        auto chunk_array = leaf_array.Slice(chunk.value_offset);
+        auto chunk_def_levels = AddIfNotNull(def_levels, chunk.level_offset);
+        auto chunk_rep_levels = AddIfNotNull(rep_levels, chunk.level_offset);
+        if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
+          ARROW_CHECK_OK(WriteArrowDictionary(chunk_def_levels, chunk_rep_levels,
+                                              chunk.levels_to_write, *chunk_array, ctx,
+                                              maybe_parent_nulls));
+        } else {
+          ARROW_CHECK_OK(WriteArrowDense(chunk_def_levels, chunk_rep_levels,
+                                         chunk.levels_to_write, *chunk_array, ctx,
+                                         maybe_parent_nulls));
+        }
+        bool is_last_chunk = i == (chunks.size() - 1);
+        if (num_buffered_values_ > 0 && !is_last_chunk) {
+          // Explicitly add a new data page according to the content-defined chunk
+          // boundaries. This way the same chunks will have the same byte-sequence
+          // in the resulting file, which can be identified by content addressible
+          // storage.
+          // Note that the last chunk doesn't trigger a new data page in order to
+          // allow subsequent WriteArrow() calls to continue writing to the same
+          // data page, the chunker's state is not being reset after the last chunk.
+          AddDataPage();

Review Comment:
   I'm not sure if I have raised the same concern in the previous review. Values of the same record cannot span two pages if we are using DataPageV2 or enabling page index. There are special logics inside `WriteArrowDictionary` and `WriteArrowDense` (see `pages_change_on_record_boundaries()`). Please make sure that it won't break.
   
   BTW, is it OK if a CDC chunk has been further split (by the column writer) into several data pages? I assume it should be fine when same writer configurations result in same pages.



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int norm_factor) {
+  if (min_chunk_size < 0) {

Review Comment:
   Can it be 0? If yes, the error message should be `non-negative`



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,

Review Comment:
   Wrap it in an anonymous namespace and remove `static`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086202719


##########
cpp/src/parquet/properties.h:
##########
@@ -245,6 +245,34 @@ class PARQUET_EXPORT ColumnProperties {
   bool page_index_enabled_;
 };
 
+// EXPERIMENTAL: Options for content-defined chunking.
+struct PARQUET_EXPORT CdcOptions {
+  /// Minimum chunk size in bytes, default is 256 KiB

Review Comment:
   Correct, but only if the max page size is smaller than the max_chunk_size. Also note that max page size counts encoded bytes while cdc max chunk size counts unencoded bytes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086659256


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1689 @@
+// Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   I think it's ok to test everything in normal builds. We should just trim the list down in Sanitizer and Valgrind builds.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2877189444

   Thanks @pitrou @wgtmac @kou @mapleFU for the reviews!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1988133292


##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1332,13 +1337,38 @@ class TypedColumnWriterImpl : public ColumnWriterImpl, public TypedColumnWriter<
       bits_buffer_->ZeroPadding();
     }
 
-    if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
-      return WriteArrowDictionary(def_levels, rep_levels, num_levels, leaf_array, ctx,
-                                  maybe_parent_nulls);
+    if (properties_->cdc_enabled()) {
+      ARROW_ASSIGN_OR_RAISE(auto boundaries,
+                            content_defined_chunker_.GetBoundaries(
+                                def_levels, rep_levels, num_levels, leaf_array));
+      for (auto chunk : boundaries) {
+        auto chunk_array = leaf_array.Slice(chunk.value_offset);
+        auto chunk_def_levels = AddIfNotNull(def_levels, chunk.level_offset);
+        auto chunk_rep_levels = AddIfNotNull(rep_levels, chunk.level_offset);
+        if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {

Review Comment:
   Great question. We are dealing with arrow arrays here, so the parquet encodings don't interfere with the chunking. However Arrow's dictionary and REE encodings do matter.
   
   "Ideally" we should fed through the "logical" values rather than the encoded ones, but that would have performance implications and I am not sure that the path builder would do the record shredding in that case. I assume the path builder only deals with the dictionary indices. 
   
   I assume that the dictionary changes are less likely, also considering the performance implications working with indices should be fine for now. I'd need to do more experimentations on this topic, so I'd prefer to defer that to a follow-up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2732291388

   @github-actions crossbow submit wheel-*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1973229135


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+

Review Comment:
   Yes, going to add the references.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1973227681


##########
python/pyarrow/_parquet.pyx:
##########
@@ -1996,6 +2001,18 @@ cdef shared_ptr[WriterProperties] _create_writer_properties(
     if dictionary_pagesize_limit is not None:
         props.dictionary_pagesize_limit(dictionary_pagesize_limit)
 
+    # content defined chunking
+    if cdc is False:
+        props.disable_cdc()
+    elif cdc is True:
+        props.enable_cdc()

Review Comment:
   Updating, `cdc` argument used to be either bool or tuple.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kou (via GitHub)" <gi...@apache.org>]

kou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2019377496


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int8_t norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes
+  int64_t target_size = avg_chunk_size - min_chunk_size;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // 3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  if (norm_factor < -3 || norm_factor > 3) {
+    ARROW_LOG(WARNING) << "norm_factor=" << std::to_string(norm_factor)
+                       << " is outside the recommended range [-3, 3]";
+  }
+
+  int mask_adjustement = 3 + norm_factor;
+  int effective_bits = mask_bits - mask_adjustement;
+  if (effective_bits == 0) {
+    return 0;
+  } else if (effective_bits > 64) {
+    throw ParquetException("The number of bits in the mask cannot exceed 64, got " +
+                           std::to_string(effective_bits));
+  } else if (effective_bits < 0) {
+    throw ParquetException(
+        "The difference between min_chunk_size=" + std::to_string(min_chunk_size) +
+        " and max_chunk_size=" + std::to_string(max_chunk_size) +
+        " is too small for the given norm_factor=" + std::to_string(norm_factor) +
+        ", either increase the size range or decrease the norm_factor.");
+  } else {
+    // create the mask by setting the top mask_bits bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int8_t norm_factor)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_factor)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(const bool value) {
+    if (chunk_size_++ < min_chunk_size_) {

Review Comment:
   Is `chunk_size_++` correct here?
   Other `Roll()` updates `chunk_size_` before comparison.



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int8_t norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes
+  int64_t target_size = avg_chunk_size - min_chunk_size;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // 3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  if (norm_factor < -3 || norm_factor > 3) {
+    ARROW_LOG(WARNING) << "norm_factor=" << std::to_string(norm_factor)
+                       << " is outside the recommended range [-3, 3]";
+  }
+
+  int mask_adjustement = 3 + norm_factor;
+  int effective_bits = mask_bits - mask_adjustement;
+  if (effective_bits == 0) {
+    return 0;
+  } else if (effective_bits > 64) {
+    throw ParquetException("The number of bits in the mask cannot exceed 64, got " +
+                           std::to_string(effective_bits));
+  } else if (effective_bits < 0) {
+    throw ParquetException(
+        "The difference between min_chunk_size=" + std::to_string(min_chunk_size) +
+        " and max_chunk_size=" + std::to_string(max_chunk_size) +
+        " is too small for the given norm_factor=" + std::to_string(norm_factor) +
+        ", either increase the size range or decrease the norm_factor.");
+  } else {
+    // create the mask by setting the top mask_bits bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int8_t norm_factor)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_factor)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(const bool value) {
+    if (chunk_size_++ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+    has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+  }
+
+  template <int ByteWidth>
+  void Roll(const uint8_t* value) {
+    // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+    // true if the hash matches the mask.
+
+    chunk_size_ += ByteWidth;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (size_t i = 0; i < ByteWidth; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  template <typename T>
+  void Roll(const T* value) {
+    return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+  }
+
+  void Roll(const uint8_t* value, int64_t length) {
+    // Update the rolling hash with a binary-like value, set has_matched_ to true if the
+    // hash matches the mask.
+
+    chunk_size_ += length;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (auto i = 0; i < length; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  bool NeedNewChunk() {
+    // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+    // set to true if we encountered a match since the last NeedNewChunk() call
+    if (ARROW_PREDICT_FALSE(has_matched_)) {
+      has_matched_ = false;
+      // in order to have a normal distribution of chunk sizes, we only create a new chunk
+      // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+      // different gearhash table (gearhash's chunk size has geometric distribution, and
+      // we use central limit theorem to approximate normal distribution, see
+      // section 6.2.1 in paper https://www.cidrdb.org/cidr2023/papers/p43-low.pdf)
+      if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+        nth_run_ = 0;
+        chunk_size_ = 0;
+        return true;
+      }
+    }
+    if (ARROW_PREDICT_FALSE(chunk_size_ >= max_chunk_size_)) {
+      // we have a hard limit on the maximum chunk size, note that we don't reset the
+      // rolling hash state here, so the next NeedNewChunk() call will continue from the
+      // current state
+      chunk_size_ = 0;
+      return true;
+    }
+    return false;
+  }
+
+  void ValidateChunks(const std::vector<Chunk>& chunks, int64_t num_levels) {

Review Comment:
   Do we need `#ifndef NDEBUG` here?
   
   ```suggestion
   #ifndef NDEBUG
     void ValidateChunks(const std::vector<Chunk>& chunks, int64_t num_levels) {
   ```



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int8_t norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes
+  int64_t target_size = avg_chunk_size - min_chunk_size;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // 3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  if (norm_factor < -3 || norm_factor > 3) {
+    ARROW_LOG(WARNING) << "norm_factor=" << std::to_string(norm_factor)

Review Comment:
   We may not need `std::string()` here:
   
   ```suggestion
       ARROW_LOG(WARNING) << "norm_factor=" << norm_factor
   ```



##########
cpp/src/parquet/properties.h:
##########
@@ -275,10 +285,60 @@ class PARQUET_EXPORT WriterProperties {
           page_checksum_enabled_(properties.page_checksum_enabled()),
           size_statistics_level_(properties.size_statistics_level()),
           sorting_columns_(properties.sorting_columns()),
-          default_column_properties_(properties.default_column_properties()) {}
+          default_column_properties_(properties.default_column_properties()),
+          content_defined_chunking_enabled_(
+              properties.content_defined_chunking_enabled()),
+          content_defined_chunking_options_(
+              properties.content_defined_chunking_options()) {}
 
     virtual ~Builder() {}
 
+    /// \brief EXPERIMENTAL: Use content-defined page chunking for all columns.
+    ///
+    /// Optimize parquet files for content addressable storage (CAS) systems by writing
+    /// data pages according to content-defined chunk boundaries. This allows for more
+    /// efficient deduplication of data across files, hence more efficient network
+    /// transfers and storage. The chunking is based on a rolling hash algorithm that
+    /// identifies chunk boundaries based on the actual content of the data.
+    Builder* enable_content_defined_chunking() {
+      content_defined_chunking_enabled_ = true;
+      return this;
+    }
+
+    /// \brief EXPERIMENTAL: Disable content-defined page chunking for all columns.
+    Builder* disable_content_defined_chunking() {
+      content_defined_chunking_enabled_ = false;
+      return this;
+    }
+
+    /// \brief EXPERIMENTAL: Specify content-defined chunking options.
+    ///
+    /// \param min_chunk_size Minimum chunk size in bytes, default 256 KiB
+    /// The rolling hash will not be updated until this size is reached for each chunk.
+    /// Note that all data sent through the hash function is counted towards the chunk
+    /// size, including definition and repetition levels if present.
+    /// \param max_chunk_size Maximum chunk size in bytes, default is 1024 KiB
+    /// The chunker will create a new chunk whenever the chunk size exceeds this value.
+    /// Note that the parquet writer has a related `pagesize` property that controls
+    /// the maximum size of a parquet data page after encoding. While setting
+    /// `pagesize` to a smaller value than `max_chunk_size` doesn't affect the
+    /// chunking effectiveness, it results in more small parquet data pages.
+    /// \param norm_factor Normalization factor to center the chunk size around the
+    /// average size more aggressively, default 0
+    /// Increasing the normalization factor increases the probability of finding a chunk,
+    /// improving the deduplication ratio, but also increasing the number of small chunks
+    /// resulting in many small parquet data pages. The default value provides a good
+    /// balance between deduplication ratio and fragmentation. Use norm_factor=1 or
+    /// norm_factor=2 to reach a higher deduplication ratio at the expense of
+    /// fragmentation.
+    Builder* content_defined_chunking_options(
+        int64_t min_chunk_size, int64_t max_chunk_size,
+        int8_t norm_factor = kDefaultCdcOptions.norm_factor) {

Review Comment:
   How about using `CDCOptions` here?
   
   ```suggestion
       Builder* content_defined_chunking_options(const CDCOptions options) {
   ```
   
   Or how about adding separated methods like `content_defined_chunking_min_chunk_size()` and `content_defined_chunking_max_chunk_size()`?
   
   They will be more extendable than this signature when we want to add a new option to `CDCOptions`.



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int8_t norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes
+  int64_t target_size = avg_chunk_size - min_chunk_size;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // 3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  if (norm_factor < -3 || norm_factor > 3) {
+    ARROW_LOG(WARNING) << "norm_factor=" << std::to_string(norm_factor)
+                       << " is outside the recommended range [-3, 3]";
+  }
+
+  int mask_adjustement = 3 + norm_factor;
+  int effective_bits = mask_bits - mask_adjustement;
+  if (effective_bits == 0) {
+    return 0;
+  } else if (effective_bits > 64) {
+    throw ParquetException("The number of bits in the mask cannot exceed 64, got " +
+                           std::to_string(effective_bits));
+  } else if (effective_bits < 0) {
+    throw ParquetException(
+        "The difference between min_chunk_size=" + std::to_string(min_chunk_size) +
+        " and max_chunk_size=" + std::to_string(max_chunk_size) +
+        " is too small for the given norm_factor=" + std::to_string(norm_factor) +
+        ", either increase the size range or decrease the norm_factor.");
+  } else {
+    // create the mask by setting the top mask_bits bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int8_t norm_factor)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_factor)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(const bool value) {
+    if (chunk_size_++ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+    has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+  }
+
+  template <int ByteWidth>
+  void Roll(const uint8_t* value) {
+    // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+    // true if the hash matches the mask.
+
+    chunk_size_ += ByteWidth;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (size_t i = 0; i < ByteWidth; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  template <typename T>
+  void Roll(const T* value) {
+    return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+  }
+
+  void Roll(const uint8_t* value, int64_t length) {
+    // Update the rolling hash with a binary-like value, set has_matched_ to true if the
+    // hash matches the mask.
+
+    chunk_size_ += length;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (auto i = 0; i < length; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  bool NeedNewChunk() {
+    // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+    // set to true if we encountered a match since the last NeedNewChunk() call
+    if (ARROW_PREDICT_FALSE(has_matched_)) {
+      has_matched_ = false;
+      // in order to have a normal distribution of chunk sizes, we only create a new chunk
+      // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+      // different gearhash table (gearhash's chunk size has geometric distribution, and
+      // we use central limit theorem to approximate normal distribution, see
+      // section 6.2.1 in paper https://www.cidrdb.org/cidr2023/papers/p43-low.pdf)
+      if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+        nth_run_ = 0;
+        chunk_size_ = 0;
+        return true;
+      }
+    }
+    if (ARROW_PREDICT_FALSE(chunk_size_ >= max_chunk_size_)) {
+      // we have a hard limit on the maximum chunk size, note that we don't reset the
+      // rolling hash state here, so the next NeedNewChunk() call will continue from the
+      // current state
+      chunk_size_ = 0;
+      return true;
+    }
+    return false;
+  }
+
+  void ValidateChunks(const std::vector<Chunk>& chunks, int64_t num_levels) {
+    // chunks must be non-empty and monotonic increasing
+    DCHECK(!chunks.empty());
+
+    // the first chunk must start at the first level
+    auto first_chunk = chunks.front();
+    DCHECK_EQ(first_chunk.level_offset, 0);
+    DCHECK_EQ(first_chunk.value_offset, 0);
+
+    // the following chunks must be contiguous, non-overlapping and monotonically
+    // increasing
+    auto sum_levels = first_chunk.levels_to_write;
+    for (size_t i = 1; i < chunks.size(); ++i) {
+      auto chunk = chunks[i];
+      auto prev_chunk = chunks[i - 1];
+      DCHECK_GT(chunk.levels_to_write, 0);
+      DCHECK_GE(chunk.value_offset, prev_chunk.value_offset);
+      DCHECK_EQ(chunk.level_offset, prev_chunk.level_offset + prev_chunk.levels_to_write);
+      sum_levels += chunk.levels_to_write;
+    }
+    DCHECK_EQ(sum_levels, num_levels);
+
+    // the last chunk must end at the last level
+    auto last_chunk = chunks.back();
+    DCHECK_EQ(last_chunk.level_offset + last_chunk.levels_to_write, num_levels);
+  }
+
+  template <typename RollFunc>
+  std::vector<Chunk> Calculate(const int16_t* def_levels, const int16_t* rep_levels,
+                               int64_t num_levels, const RollFunc& RollValue) {
+    // Calculate the chunk boundaries for typed Arrow arrays.
+    std::vector<Chunk> chunks;
+    int64_t offset;
+    int64_t prev_offset = 0;
+    int64_t prev_value_offset = 0;
+    bool has_def_levels = level_info_.def_level > 0;
+    bool has_rep_levels = level_info_.rep_level > 0;
+
+    if (!has_rep_levels && !has_def_levels) {
+      // fastest path for non-nested non-null data
+      for (offset = 0; offset < num_levels; ++offset) {
+        RollValue(offset);
+        if (NeedNewChunk()) {
+          chunks.push_back({prev_offset, prev_offset, offset - prev_offset});
+          prev_offset = offset;
+        }
+      }
+      // set the previous value offset to add the last chunk
+      prev_value_offset = prev_offset;
+    } else if (!has_rep_levels) {
+      // non-nested data with nulls
+      int16_t def_level;
+      for (int64_t offset = 0; offset < num_levels; ++offset) {
+        def_level = def_levels[offset];
+
+        Roll(&def_level);
+        if (def_level == level_info_.def_level) {
+          RollValue(offset);
+        }
+        if (NeedNewChunk()) {
+          chunks.push_back({prev_offset, prev_offset, offset - prev_offset});
+          prev_offset = offset;
+        }
+      }
+      // set the previous value offset to add the last chunk
+      prev_value_offset = prev_offset;
+    } else {
+      // nested data with nulls
+      int16_t def_level;
+      int16_t rep_level;
+      int64_t value_offset = 0;
+
+      for (offset = 0; offset < num_levels; ++offset) {
+        def_level = def_levels[offset];
+        rep_level = rep_levels[offset];
+
+        Roll(&def_level);
+        Roll(&rep_level);
+        if (def_level == level_info_.def_level) {
+          RollValue(value_offset);
+        }
+
+        if ((rep_level == 0) && NeedNewChunk()) {
+          // if we are at a record boundary and need a new chunk, we create a new chunk
+          auto levels_to_write = offset - prev_offset;
+          if (levels_to_write > 0) {
+            chunks.push_back({prev_offset, prev_value_offset, levels_to_write});
+            prev_offset = offset;
+            prev_value_offset = value_offset;
+          }
+        }
+        if (def_level >= level_info_.repeated_ancestor_def_level) {
+          // we only increment the value offset if we have a leaf value
+          ++value_offset;
+        }
+      }

Review Comment:
   Do we need the following that exists in other clauses here too?
   
   ```cpp
         // set the previous value offset to add the last chunk
         prev_value_offset = prev_offset;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2760716284

   @github-actions crossbow submit wheel-*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2739878882

   @kou do you know what could be the problem with the MinGW unity build https://github.com/apache/arrow/actions/runs/13966289727/job/39097296013?pr=45360#step:7:818 ? I can think of a missing header or a namespace problem but hard to figure out for that build.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2619243188

   > Is cdc a part of the parquet spec? Or is it a poc?
   
   It is not. You can think of it as a feature similar to the existing options to specify how record batches and pages are being split. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1973999507


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }

Review Comment:
   Checked on godbolt with `-O3` and your version is indeed better:
   
   Source:
   
   ```cpp
   #include <stdint.h>
   
   bool has_matched_ = false;
   uint64_t rolling_hash_ = 0;
   
   void Roll(const uint64_t value) {
     rolling_hash_ = (rolling_hash_ << 1) + value;
     if ((rolling_hash_ & 0xffff000000000000) == 0) {
       has_matched_ = true;
     }
   }
   ```
   
   Assembly:
   
   ```asm
   Roll(unsigned long):
           mov     rax, QWORD PTR rolling_hash_[rip]
           lea     rax, [rdi+rax*2]
           mov     QWORD PTR rolling_hash_[rip], rax
           shr     rax, 48
           jne     .L1
           mov     BYTE PTR has_matched_[rip], 1
   .L1:
           ret
   rolling_hash_:
           .zero   8
   has_matched_:
           .zero   1
   ```
   
   Source:
   
   ```cpp
   #include <stdint.h>
   
   bool has_matched_ = false;
   uint64_t rolling_hash_ = 0;
   
   void Roll(const uint64_t value) {
     rolling_hash_ = (rolling_hash_ << 1) + value;
     has_matched_ = has_matchd_ || ((rolling_hash_ & 0xffff000000000000) == 0);
   }
   ```
   
   Assembly:
   
   ```asm
   Roll(unsigned long):
           mov     rax, QWORD PTR rolling_hash_[rip]
           lea     rax, [rdi+rax*2]
           mov     QWORD PTR rolling_hash_[rip], rax
           shr     rax, 48
           sete    al
           or      BYTE PTR has_matched_[rip], al
           ret
   rolling_hash_:
           .zero   8
   has_matched_:
           .zero   1
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025198500


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027827840


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();
+    auto min_length = ElementCount(min_chunk_size, byte_width, nullable);
+    auto max_length = ElementCount(max_chunk_size, byte_width, nullable);
+    AssertAllBetween(base_info.page_lengths, min_length, max_length);
+    AssertAllBetween(modified_info.page_lengths, min_length, max_length);
+  } else if (::arrow::is_base_binary_like(dtype->id()) && !enable_dictionary) {
+    AssertAllBetween(base_info.page_sizes, min_chunk_size, max_chunk_size);
+    AssertAllBetween(modified_info.page_sizes, min_chunk_size, max_chunk_size);
+  }
+}
+
+constexpr int64_t kMinChunkSize = 8 * 1024;
+constexpr int64_t kMaxChunkSize = 32 * 1024;
+constexpr int64_t kPartSize = 128 * 1024;
+constexpr int64_t kEditSize = 128;
+
+struct CaseConfig {
+  // Arrow data type to generate the testing data for
+  std::shared_ptr<::arrow::DataType> dtype;
+  // Whether the data type is nullable
+  bool is_nullable;
+  // Approximate number of bytes per record to calculate the number of elements to
+  // generate
+  size_t bytes_per_record;
+  // Data page version to use
+  ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1;
+};
+
+// Define PrintTo for MyStruct
+void PrintTo(const CaseConfig& param, std::ostream* os) {
+  *os << "{ " << param.dtype->ToString();
+  if (param.is_nullable) {
+    *os << " nullable";
+  }
+  *os << " }";
+}
+
+class TestCDC : public ::testing::Test {
+ public:
+  uint64_t GetMask(const ContentDefinedChunker& cdc) const { return cdc.GetMask(); }
+};
+
+TEST_F(TestCDC, RollingHashMaskCalculation) {
+  auto le = LevelInfo();
+  auto min_size = 256 * 1024;
+  auto max_size = 1024 * 1024;
+
+  auto cdc0 = ContentDefinedChunker(le, min_size, max_size, 0);
+  ASSERT_EQ(GetMask(cdc0), 0xFFFE000000000000);
+
+  auto cdc1 = ContentDefinedChunker(le, min_size, max_size, 1);
+  ASSERT_EQ(GetMask(cdc1), 0xFFFC000000000000);
+
+  auto cdc2 = ContentDefinedChunker(le, min_size, max_size, 2);
+  ASSERT_EQ(GetMask(cdc2), 0xFFF8000000000000);
+
+  auto cdc3 = ContentDefinedChunker(le, min_size, max_size, 3);
+  ASSERT_EQ(GetMask(cdc3), 0xFFF0000000000000);
+
+  auto cdc4 = ContentDefinedChunker(le, min_size, max_size, -1);
+  ASSERT_EQ(GetMask(cdc4), 0xFFFF000000000000);
+
+  // this is the smallest possible mask always matching, by using 8 hashtables
+  // we are going to have a match every 8 bytes; this is an unrealistic case
+  // but checking for the correctness of the mask calculation
+  auto cdc5 = ContentDefinedChunker(le, 0, 16, 0);
+  ASSERT_EQ(GetMask(cdc5), 0x0000000000000000);
+
+  auto cdc6 = ContentDefinedChunker(le, 0, 32, 1);
+  ASSERT_EQ(GetMask(cdc6), 0x0000000000000000);
+
+  auto cdc7 = ContentDefinedChunker(le, 0, 16, -1);
+  ASSERT_EQ(GetMask(cdc7), 0x8000000000000000);
+
+  // another unrealistic case, checking for the validation
+  auto cdc8 = ContentDefinedChunker(le, 128, 384, -60);
+  ASSERT_EQ(GetMask(cdc8), 0xFFFFFFFFFFFFFFFF);
+}
+
+TEST_F(TestCDC, WriteSingleColumnParquetFile) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  std::vector<int32_t> numbers = {1, 2, 3, 4, 5};
+  std::vector<uint8_t> valid_bits = {1, 0, 1, 0, 1};
+  EXPECT_THROW(
+      int_column_writer.WriteBatch(numbers.size(), nullptr, nullptr, numbers.data()),
+      ParquetException);
+  EXPECT_THROW(int_column_writer.WriteBatchSpaced(numbers.size(), nullptr, nullptr,
+                                                  valid_bits.data(), 0, numbers.data()),
+               ParquetException);
+}
+
+TEST_F(TestCDC, LastChunkDoesntTriggerAddDataPage) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()
+                   ->content_defined_chunking_options(kMinChunkSize, kMaxChunkSize, 0)
+                   ->disable_dictionary()
+                   ->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  ASSERT_OK_AND_ASSIGN(auto array, ::arrow::gen::Step()->Generate(8000));
+  auto arrow_props = default_arrow_writer_properties();
+  auto arrow_ctx = ArrowWriteContext(default_memory_pool(), arrow_props.get());
+
+  // Calling WriteArrow twice, we expect that the first call doesn't add a new data page
+  // at the end allowing subsequent calls to append to the same page
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+
+  int_column_writer.Close();
+  writer->Close();
+  ASSERT_OK_AND_ASSIGN(auto buffer, sink->Finish());
+
+  auto info = GetColumnParquetInfo(buffer);
+  ASSERT_EQ(info.size(), 1);
+
+  // AssertAllBetween allow the last chunk size to be smaller than the min_chunk_size
+  AssertAllBetween(info[0].page_sizes, kMinChunkSize, kMaxChunkSize);
+  AssertAllBetween(info[0].page_lengths, 3000, 5000);
+}
+
+TEST_F(TestCDC, ChunkSizeParameterValidation) {
+  // Test that constructor validates min/max chunk size parameters
+  auto li = LevelInfo();
+
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 256 * 1024, 1024 * 1024));
+
+  // with norm_factor=0 the difference between min and max chunk size must be
+  // at least 16
+  ASSERT_THROW(ContentDefinedChunker(li, 0, -1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1024, 512), ParquetException);
+
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16));
+  ASSERT_THROW(ContentDefinedChunker(li, -16, -16), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 16, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 32, 32), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 32, 48));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1024 * 1024, 2 * 1024 * 1024));
+  ASSERT_NO_THROW(
+      ContentDefinedChunker(li, 1024 * 1024 * 1024L, 2LL * 1024 * 1024 * 1024L));
+
+  // with norm_factor=1 the difference between min and max chunk size must be
+  // at least 64
+  ASSERT_THROW(ContentDefinedChunker(li, 1, -1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 32, 1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1, 33, 1));
+
+  // with norm_factor=2 the difference between min and max chunk size must be
+  // at least 128
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 63, 2), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 64, 2));
+
+  // with norm_factor=-1 the difference between min and max chunk size must be
+  // at least 8
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 7, -1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 8, -1));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16, -1));
+
+  // test the norm_factor extremes
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0, -68), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 1, -67));
+  ASSERT_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 59),
+               ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 58));
+}
+
+class TestCDCSingleRowGroup : public ::testing::TestWithParam<CaseConfig> {
+ protected:
+  // Column random table parts for testing
+  std::shared_ptr<Field> field_;
+  std::shared_ptr<Table> part1_, part2_, part3_, part4_, part5_, part6_, part7_;
+
+  void SetUp() override {
+    const auto& param = GetParam();
+    auto field_ = ::arrow::field("f0", param.dtype, param.is_nullable);
+    auto schema = ::arrow::schema({field_});
+
+    auto part_length = kPartSize / param.bytes_per_record;
+    auto edit_length = kEditSize / param.bytes_per_record;
+    ASSERT_OK_AND_ASSIGN(part1_, GenerateTable(schema, part_length, 0));
+    ASSERT_OK_AND_ASSIGN(part2_, GenerateTable(schema, edit_length, 1));
+    ASSERT_OK_AND_ASSIGN(part3_, GenerateTable(schema, part_length, part_length));
+    ASSERT_OK_AND_ASSIGN(part4_, GenerateTable(schema, edit_length, 2));
+    ASSERT_OK_AND_ASSIGN(part5_, GenerateTable(schema, part_length, 2 * part_length));
+    ASSERT_OK_AND_ASSIGN(part6_, GenerateTable(schema, edit_length, 3));
+    ASSERT_OK_AND_ASSIGN(part7_, GenerateTable(schema, edit_length, 4));
+  }
+};
+
+TEST_P(TestCDCSingleRowGroup, DeleteOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }

Review Comment:
   I made it more robust adding the number of `null` / `empty list` elements to the number of values. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027838874


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();

Review Comment:
   I reworked this estimation to properly account for nulls as well. Added a comment ti highlight that the CDC chunker increments the chunk size by 1 for each boolean element.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025168198


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025777377


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)

Review Comment:
   Need to rewire the functions a little bit to pass the array instead of the dtype.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025138659


##########
cpp/src/parquet/properties.h:
##########
@@ -245,6 +245,14 @@ class PARQUET_EXPORT ColumnProperties {
   bool page_index_enabled_;
 };
 
+struct CDCOptions {

Review Comment:
   Updated, moved the field docstrings here from the `content_defined_chunking_options()` method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1985090207


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,250 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_hashtable.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][bytes[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling
+    // hash state here, so the next NeedNewChunk() call will continue from the current
+    // state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename T>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const T& leaf_array) {
+  std::vector<Chunk> chunks;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      chunks.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(def_levels[offset]);
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      chunks.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else {
+    // nested data with nulls
+    bool has_leaf_value;
+    bool is_record_boundary;
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t value_offset = 0;
+    int64_t record_level_offset = 0;
+    int64_t record_value_offset = 0;
+
+    for (int64_t level_offset = 0; level_offset < num_levels; ++level_offset) {
+      def_level = def_levels[level_offset];
+      rep_level = rep_levels[level_offset];
+
+      has_leaf_value = def_level >= level_info_.repeated_ancestor_def_level;
+      is_record_boundary = rep_level == 0;
+
+      Roll(def_level);
+      Roll(rep_level);
+      if (has_leaf_value) {
+        Roll(leaf_array.GetView(value_offset));
+      }
+
+      if (is_record_boundary && NeedNewChunk()) {
+        auto levels_to_write = level_offset - record_level_offset;
+        if (levels_to_write > 0) {
+          chunks.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+          record_level_offset = level_offset;
+          record_value_offset = value_offset;
+        }
+      }
+
+      if (has_leaf_value) {
+        ++value_offset;
+      }
+    }
+
+    auto levels_to_write = num_levels - record_level_offset;
+    if (levels_to_write > 0) {
+      chunks.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+    }
+  }
+
+  return chunks;
+}
+
+#define PRIMITIVE_CASE(TYPE_ID, ArrowType)               \
+  case ::arrow::Type::TYPE_ID:                           \
+    return Calculate(def_levels, rep_levels, num_levels, \
+                     static_cast<const ::arrow::ArrowType##Array&>(values));
+

Review Comment:
   I updated the implementation to use the same code path for same width types, also switched to use the raw buffer pointers. I introduced a `RollValue` closure to handle different paths. Please take a look again.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1983556086


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \

Review Comment:
   Good point, it was messier in the beginning. Updating.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1985418603


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, uint64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal128Builder, ::arrow::Decimal128(val % max_value))
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal256Builder, ::arrow::Decimal256(val % max_value))
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      GENERATE_CASE_BODY(::arrow::FixedSizeBinaryBuilder,
+                         std::string("bin_") + std::to_string(val).substr(0, size - 4))
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, uint64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(const std::shared_ptr<Table>& table,
+                                                   uint64_t min_chunk_size,
+                                                   uint64_t max_chunk_size,
+                                                   bool enable_dictionary = false,
+
+                                                   int64_t row_group_size = 1024 * 1024) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_cdc()
+      ->cdc_size_range(min_chunk_size, max_chunk_size)
+      ->cdc_norm_factor(0);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+struct PageSizes {
+  std::vector<uint64_t> lengths;
+  std::vector<uint64_t> sizes;
+};
+
+PageSizes GetColumnPageSizes(const std::shared_ptr<Buffer>& data, int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  PageSizes result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        result.sizes.push_back(data_page->size());
+        result.lengths.push_back(data_page->num_values());
+      }
+    }
+  }
+
+  return result;
+}
+
+Result<PageSizes> WriteAndGetPageSizes(const std::shared_ptr<Table>& table,
+                                       uint64_t min_chunk_size, uint64_t max_chunk_size,
+                                       bool enable_dictionary = false,
+                                       int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnPageSizes(buffer, column_index);
+}
+
+void AssertAllBetween(const std::vector<uint64_t>& values, uint64_t min, uint64_t max,
+                      bool expect_dictionary_fallback = false) {
+  // expect the last chunk since it is not guaranteed to be within the range
+  if (expect_dictionary_fallback) {
+    // if dictionary encoding is enabled, the writer can fallback to plain
+    // encoding splitting within a content defined chunk, so we can't
+    // guarantee that all chunks are within the range in this case, but we
+    // know that there can be at most 2 pages smaller than the min_chunk_size
+    size_t smaller_count = 0;
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      if (values[i] < min) {
+        smaller_count++;
+      } else {
+        ASSERT_LE(values[i], max);
+      }
+    }
+    ASSERT_LE(smaller_count, 2);
+  } else {
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      ASSERT_GE(values[i], min);
+      ASSERT_LE(values[i], max);
+    }
+  }
+  ASSERT_LE(values.back(), max);
+}
+
+std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>> FindDifferences(
+    const std::vector<uint64_t>& first, const std::vector<uint64_t>& second) {
+  // Compute LCS table.
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack to get common indices.
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      i--;
+    } else {
+      j--;
+    }
+  }
+  std::reverse(common.begin(), common.end());
+
+  // Build raw differences.
+  std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Merge adjacent diffs if one side is empty in the first diff and the other side
+  // is empty in the next diff, to avoid splitting single changes into two parts.
+  std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+      if (can_merge_a) {
+        // Combine into one change
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    merged.push_back(std::move(diff));
+  }
+
+  return merged;
+}
+
+void PrintDifferences(
+    const std::vector<uint64_t>& original, const std::vector<uint64_t>& modified,
+    std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>>& diffs) {
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  std::vector<uint64_t> first = {1, 2, 3, 4, 5};
+  std::vector<uint64_t> second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({2, 3}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  std::vector<uint64_t> first = {1, 2, 3, 4, 5, 6, 7};
+  std::vector<uint64_t> second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({2, 3}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, std::vector<uint64_t>({5}));
+  ASSERT_EQ(diffs[1].second, std::vector<uint64_t>({10}));
+
+  ASSERT_EQ(diffs[2].first, std::vector<uint64_t>({7}));
+  ASSERT_EQ(diffs[2].second, std::vector<uint64_t>({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  std::vector<uint64_t> first = {1, 2, 3};
+  std::vector<uint64_t> second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  std::vector<uint64_t> first = {};
+  std::vector<uint64_t> second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  std::vector<uint64_t> first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  std::vector<uint64_t> second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  std::vector<uint64_t> first = {2169, 1976, 2180, 2147, 1934, 1772,
+                                 1914, 2075, 2154, 1940, 1934, 1970};
+  std::vector<uint64_t> second = {2169, 1976, 2180, 2147, 2265, 1804,
+                                  1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  std::vector<uint64_t> original = {445, 312, 393, 401, 410, 138, 558, 457};
+  std::vector<uint64_t> modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({401}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertUpdateCase(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const std::vector<uint64_t>& original,
+                      const std::vector<uint64_t>& modified, uint8_t n_modifications) {
+  auto diffs = FindDifferences(original, modified);
+  if (diffs.size() > n_modifications) {
+    PrintDifferences(original, modified, diffs);
+  }
+  ASSERT_LE(diffs.size(), n_modifications);
+
+  for (const auto& diff : diffs) {
+    if (!::arrow::is_list_like(dtype->id())) {
+      uint64_t left_sum = 0, right_sum = 0;
+      for (const auto& val : diff.first) left_sum += val;
+      for (const auto& val : diff.second) right_sum += val;
+      ASSERT_EQ(left_sum, right_sum);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original == modified);
+  }
+}
+
+void AssertDeleteCase(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const std::vector<uint64_t>& original,
+                      const std::vector<uint64_t>& modified, uint8_t n_modifications,
+                      uint64_t edit_length) {
+  auto diffs = FindDifferences(original, modified);
+  if (diffs.size() != n_modifications) {
+    PrintDifferences(original, modified, diffs);
+  }
+  ASSERT_EQ(diffs.size(), n_modifications);
+
+  for (const auto& diff : diffs) {
+    if (!::arrow::is_list_like(dtype->id())) {
+      uint64_t left_sum = 0, right_sum = 0;
+      for (const auto& val : diff.first) left_sum += val;
+      for (const auto& val : diff.second) right_sum += val;
+      ASSERT_EQ(left_sum, right_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+}
+
+void AssertInsertCase(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const std::vector<uint64_t>& original,
+                      const std::vector<uint64_t>& modified, uint8_t n_modifications,
+                      uint64_t edit_length) {
+  auto diffs = FindDifferences(original, modified);
+  if (diffs.size() != n_modifications) {
+    PrintDifferences(original, modified, diffs);
+  }
+  ASSERT_EQ(diffs.size(), n_modifications);
+
+  for (const auto& diff : diffs) {
+    if (!::arrow::is_list_like(dtype->id())) {
+      uint64_t left_sum = 0, right_sum = 0;
+      for (const auto& val : diff.first) left_sum += val;
+      for (const auto& val : diff.second) right_sum += val;
+      ASSERT_EQ(left_sum + edit_length, right_sum);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+}
+
+void AssertAppendCase(const std::vector<uint64_t>& original,
+                      const std::vector<uint64_t>& modified) {
+  ASSERT_GE(modified.size(), original.size());
+  for (size_t i = 0; i < original.size() - 1; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  ASSERT_GT(modified[original.size() - 1], original.back());
+}
+
+uint64_t ElementCount(uint64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      PageSizes base_result, PageSizes modified_result, bool nullable,
+                      bool enable_dictionary, uint64_t min_chunk_size,
+                      uint64_t max_chunk_size) {
+  max_chunk_size = static_cast<uint64_t>(max_chunk_size * 1.2);
+  if (::arrow::is_fixed_width(dtype->id())) {
+    auto min_length = ElementCount(min_chunk_size, dtype->byte_width(), nullable);
+    auto max_length = ElementCount(max_chunk_size, dtype->byte_width(), nullable);
+    AssertAllBetween(base_result.lengths, min_length, max_length,
+                     /*expect_dictionary_fallback=*/enable_dictionary);
+    AssertAllBetween(modified_result.lengths, min_length, max_length,
+                     /*expect_dictionary_fallback=*/enable_dictionary);
+  } else if (::arrow::is_base_binary_like(dtype->id()) && !nullable &&
+             !enable_dictionary) {
+    AssertAllBetween(base_result.sizes, min_chunk_size, max_chunk_size);
+    AssertAllBetween(modified_result.sizes, min_chunk_size, max_chunk_size);
+  }
+}
+
+constexpr uint64_t kMinChunkSize = 8 * 1024;
+constexpr uint64_t kMaxChunkSize = 32 * 1024;
+constexpr uint64_t kPartSize = 128 * 1024;
+constexpr uint64_t kEditSize = 128;
+
+class TestColumnCDC : public ::testing::TestWithParam<
+                          std::tuple<std::shared_ptr<::arrow::DataType>, bool, size_t>> {

Review Comment:
   Updated to use a struct.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1983591472


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {

Review Comment:
   With `norm_factor=2`
   
   ![newplot-12](https://github.com/user-attachments/assets/3ba62312-fb85-4a2b-b704-dea0ff75d1fc)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1985565370


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }

Review Comment:
   This is called as cut-point skipping in the [FastCDC paper](https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf):
   
   > Our third observation suggests that the predefined
   minimum chunk size used to avoid generating the very
   small-sized chunks ( e.g., LBFS [25] employs the mini-
   mum chunk size of 2KB for Rabin-based CDC) can be
   employed for cut-point skipping during CDC, i.e., judi-
   ciously skipping some identified cut-points to eliminate
   the CDC operations in this region. Enlarging this mini-
   mum chunk size can further speed up the chunking pro-
   cess but at the cost of decreasing the deduplication ra-
   tio. This is because many chunks with skipped cut-points
   are not divided truly according to the data contents (i.e.,
   content-defined).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1992220062


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));

Review Comment:
   Sure, updating.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1992187672


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             int64_t min_size, int64_t max_size,
+                                             int8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(min_size),
+      max_size_(max_size),
+      hash_mask_(GetMask(min_size, max_size, norm_factor)) {
+  if (min_size_ < 0) {
+    throw ParquetException("min_size must be non-negative");
+  }
+  if (max_size_ < 0) {
+    throw ParquetException("max_size must be non-negative");
+  }
+  if (min_size_ > max_size_) {
+    throw ParquetException("min_size must be less than or equal to max_size");
+  }
+}
+
+void ContentDefinedChunker::Roll(const bool value) {
+  if (chunk_size_++ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+  has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+}
+
+template <int ByteWidth>
+void ContentDefinedChunker::Roll(const uint8_t* value) {
+  chunk_size_ += ByteWidth;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (size_t i = 0; i < ByteWidth; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T* value) {
+  return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+}
+
+void ContentDefinedChunker::Roll(const uint8_t* value, int64_t length) {
+  chunk_size_ += length;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (auto i = 0; i < length; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, note that we don't reset the
+    // rolling hash state here, so the next NeedNewChunk() call will continue from the
+    // current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename RollFunc>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const RollFunc& RollValue) {
+  std::vector<Chunk> chunks;
+  int64_t offset;
+  int64_t prev_offset = 0;
+  int64_t prev_value_offset = 0;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    for (offset = 0; offset < num_levels; ++offset) {
+      RollValue(offset);
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int16_t def_level;
+    for (int64_t offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+
+      Roll(&def_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(offset);
+      }
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else {
+    // nested data with nulls
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t value_offset = 0;
+
+    for (offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+      rep_level = rep_levels[offset];
+
+      Roll(&def_level);
+      Roll(&rep_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(value_offset);
+      }
+
+      if ((rep_level == 0) && NeedNewChunk()) {
+        // if we are at a record boundary and need a new chunk, we create a new chunk
+        auto levels_to_write = offset - prev_offset;
+        if (levels_to_write > 0) {
+          chunks.emplace_back(prev_offset, prev_value_offset, levels_to_write);
+          prev_offset = offset;
+          prev_value_offset = value_offset;
+        }
+      }
+      if (def_level >= level_info_.repeated_ancestor_def_level) {
+        // we only increment the value offset if we have a leaf value
+        ++value_offset;
+      }
+    }
+  }
+
+  // add the last chunk if we have any levels left
+  if (prev_offset < num_levels) {
+    chunks.emplace_back(prev_offset, prev_value_offset, num_levels - prev_offset);
+  }
+  return chunks;
+}
+
+#define FIXED_WIDTH_CASE(ByteWidth)                                       \
+  {                                                                       \
+    const auto raw_values = values.data()->GetValues<uint8_t>(1);         \
+    return Calculate(def_levels, rep_levels, num_levels, [&](int64_t i) { \
+      return Roll<ByteWidth>(raw_values + i * ByteWidth);                 \
+    });                                                                   \
+  }
+
+#define BINARY_LIKE_CASE(ArrayType)                                       \
+  {                                                                       \
+    const auto& array = static_cast<const ArrayType&>(values);            \
+    const uint8_t* value;                                                 \
+    ArrayType::offset_type length;                                        \
+    return Calculate(def_levels, rep_levels, num_levels, [&](int64_t i) { \
+      value = array.GetValue(i, &length);                                 \
+      Roll(value, length);                                                \
+    });                                                                   \
+  }
+
+const std::vector<Chunk> ContentDefinedChunker::GetBoundaries(
+    const int16_t* def_levels, const int16_t* rep_levels, int64_t num_levels,
+    const ::arrow::Array& values) {
+  auto type_id = values.type()->id();
+  switch (type_id) {
+    case ::arrow::Type::NA: {
+      return Calculate(def_levels, rep_levels, num_levels, [](int64_t) {});
+    }
+    case ::arrow::Type::BOOL: {
+      const auto& bool_array = static_cast<const ::arrow::BooleanArray&>(values);
+      return Calculate(def_levels, rep_levels, num_levels,
+                       [&](int64_t i) { return Roll(bool_array.Value(i)); });
+    }
+    case ::arrow::Type::INT8:
+    case ::arrow::Type::UINT8:
+      FIXED_WIDTH_CASE(1)
+    case ::arrow::Type::INT16:
+    case ::arrow::Type::UINT16:
+    case ::arrow::Type::HALF_FLOAT:
+      FIXED_WIDTH_CASE(2)
+    case ::arrow::Type::INT32:
+    case ::arrow::Type::UINT32:
+    case ::arrow::Type::FLOAT:
+    case ::arrow::Type::DATE32:
+    case ::arrow::Type::TIME32:
+      FIXED_WIDTH_CASE(4)
+    case ::arrow::Type::INT64:
+    case ::arrow::Type::UINT64:
+    case ::arrow::Type::DOUBLE:
+    case ::arrow::Type::DATE64:
+    case ::arrow::Type::TIME64:
+    case ::arrow::Type::TIMESTAMP:
+    case ::arrow::Type::DURATION:
+      FIXED_WIDTH_CASE(8)
+    case ::arrow::Type::DECIMAL128:

Review Comment:
   Apparently decimal32 and decimal64 are not supported yet by the arrow parquet implementation:
   
   ```
   Failed
   '_error_or_value60.status()' failed with NotImplemented: Unhandled type for Arrow to Parquet schema conversion: decimal64(18, 6)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2760721387

   @github-actions crossbow submit -g packaging


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2019759762


##########
cpp/src/parquet/properties.h:
##########
@@ -275,10 +285,60 @@ class PARQUET_EXPORT WriterProperties {
           page_checksum_enabled_(properties.page_checksum_enabled()),
           size_statistics_level_(properties.size_statistics_level()),
           sorting_columns_(properties.sorting_columns()),
-          default_column_properties_(properties.default_column_properties()) {}
+          default_column_properties_(properties.default_column_properties()),
+          content_defined_chunking_enabled_(
+              properties.content_defined_chunking_enabled()),
+          content_defined_chunking_options_(
+              properties.content_defined_chunking_options()) {}
 
     virtual ~Builder() {}
 
+    /// \brief EXPERIMENTAL: Use content-defined page chunking for all columns.
+    ///
+    /// Optimize parquet files for content addressable storage (CAS) systems by writing
+    /// data pages according to content-defined chunk boundaries. This allows for more
+    /// efficient deduplication of data across files, hence more efficient network
+    /// transfers and storage. The chunking is based on a rolling hash algorithm that
+    /// identifies chunk boundaries based on the actual content of the data.
+    Builder* enable_content_defined_chunking() {
+      content_defined_chunking_enabled_ = true;
+      return this;
+    }
+
+    /// \brief EXPERIMENTAL: Disable content-defined page chunking for all columns.
+    Builder* disable_content_defined_chunking() {
+      content_defined_chunking_enabled_ = false;
+      return this;
+    }
+
+    /// \brief EXPERIMENTAL: Specify content-defined chunking options.
+    ///
+    /// \param min_chunk_size Minimum chunk size in bytes, default 256 KiB
+    /// The rolling hash will not be updated until this size is reached for each chunk.
+    /// Note that all data sent through the hash function is counted towards the chunk
+    /// size, including definition and repetition levels if present.
+    /// \param max_chunk_size Maximum chunk size in bytes, default is 1024 KiB
+    /// The chunker will create a new chunk whenever the chunk size exceeds this value.
+    /// Note that the parquet writer has a related `pagesize` property that controls
+    /// the maximum size of a parquet data page after encoding. While setting
+    /// `pagesize` to a smaller value than `max_chunk_size` doesn't affect the
+    /// chunking effectiveness, it results in more small parquet data pages.
+    /// \param norm_factor Normalization factor to center the chunk size around the
+    /// average size more aggressively, default 0
+    /// Increasing the normalization factor increases the probability of finding a chunk,
+    /// improving the deduplication ratio, but also increasing the number of small chunks
+    /// resulting in many small parquet data pages. The default value provides a good
+    /// balance between deduplication ratio and fragmentation. Use norm_factor=1 or
+    /// norm_factor=2 to reach a higher deduplication ratio at the expense of
+    /// fragmentation.
+    Builder* content_defined_chunking_options(
+        int64_t min_chunk_size, int64_t max_chunk_size,
+        int8_t norm_factor = kDefaultCdcOptions.norm_factor) {

Review Comment:
   I don't have a strong opinion. Previously I used separate methods but @pitrou requested to change to a struct instead. I didn't want to expose CDCOptions neither to implement a builder for it since the validation logic is placed in the chunker module, hence I used arguments instead. 
   
   > They will be more extendable than this signature when we want to add a new option to CDCOptions.
   
   I can switch to it, though I don't expect more parameters to be there. I also marked this method as experimental so we can more easily break it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2863138655

   @wgtmac @pitrou gentle ping :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086220073


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1689 @@
+// Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   Just for the record: these tests take 20s here in plain debug mode. This is ok, but they will be much longer on sanitizer and Valgrind builds (see [here](https://github.com/apache/arrow/actions/runs/14980253105/job/42082443677?pr=45360#step:7:7536): 200s on the ASAN+UBSAN build). Can we arrange to make them shorter on those builds, perhaps reducing the number of iterations?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025161024


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());

Review Comment:
   Updated everywhere.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025780959


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();

Review Comment:
   This is an implementation detail because the roll hash is calculated for each boolean value. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027824354


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);

Review Comment:
   Removed these checks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1998839038


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             int64_t min_size, int64_t max_size,
+                                             int8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(min_size),
+      max_size_(max_size),
+      hash_mask_(GetMask(min_size, max_size, norm_factor)) {
+  if (min_size_ < 0) {
+    throw ParquetException("min_size must be non-negative");
+  }
+  if (max_size_ < 0) {
+    throw ParquetException("max_size must be non-negative");
+  }
+  if (min_size_ > max_size_) {
+    throw ParquetException("min_size must be less than or equal to max_size");
+  }
+}
+
+void ContentDefinedChunker::Roll(const bool value) {
+  if (chunk_size_++ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+  has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+}
+
+template <int ByteWidth>
+void ContentDefinedChunker::Roll(const uint8_t* value) {
+  chunk_size_ += ByteWidth;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (size_t i = 0; i < ByteWidth; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T* value) {
+  return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+}
+
+void ContentDefinedChunker::Roll(const uint8_t* value, int64_t length) {
+  chunk_size_ += length;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (auto i = 0; i < length; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, note that we don't reset the
+    // rolling hash state here, so the next NeedNewChunk() call will continue from the
+    // current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename RollFunc>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const RollFunc& RollValue) {
+  std::vector<Chunk> chunks;
+  int64_t offset;
+  int64_t prev_offset = 0;
+  int64_t prev_value_offset = 0;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    for (offset = 0; offset < num_levels; ++offset) {
+      RollValue(offset);
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int16_t def_level;
+    for (int64_t offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+
+      Roll(&def_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(offset);
+      }
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else {
+    // nested data with nulls
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t value_offset = 0;
+
+    for (offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+      rep_level = rep_levels[offset];
+
+      Roll(&def_level);
+      Roll(&rep_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(value_offset);
+      }
+
+      if ((rep_level == 0) && NeedNewChunk()) {
+        // if we are at a record boundary and need a new chunk, we create a new chunk
+        auto levels_to_write = offset - prev_offset;
+        if (levels_to_write > 0) {
+          chunks.emplace_back(prev_offset, prev_value_offset, levels_to_write);
+          prev_offset = offset;
+          prev_value_offset = value_offset;
+        }
+      }
+      if (def_level >= level_info_.repeated_ancestor_def_level) {
+        // we only increment the value offset if we have a leaf value
+        ++value_offset;
+      }
+    }
+  }
+
+  // add the last chunk if we have any levels left
+  if (prev_offset < num_levels) {
+    chunks.emplace_back(prev_offset, prev_value_offset, num_levels - prev_offset);
+  }
+  return chunks;
+}
+
+#define FIXED_WIDTH_CASE(ByteWidth)                                       \
+  {                                                                       \
+    const auto raw_values = values.data()->GetValues<uint8_t>(1);         \

Review Comment:
   Updated, going to add tests using array offsets.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086711091


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+namespace {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_level parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_level Normalization level (default 0)
+// @return The mask used to compare against the rolling hash
+uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size, int norm_level) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be non-negative");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes; also divide by the number of gearhash tables to have a
+  // a more gaussian-like distribution
+  int64_t target_size = (avg_chunk_size - min_chunk_size) / kNumGearhashTables;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // a user defined `norm_level` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_level we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_level is 0 by
+  // default
+  int effective_bits = mask_bits - norm_level;
+
+  if (effective_bits < 1 || effective_bits > 63) {
+    throw ParquetException(
+        "The number of bits in the CDC mask must be between 1 and 63, got " +
+        std::to_string(effective_bits));
+  } else {
+    // create the mask by setting the top bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+}  // namespace
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int norm_level)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_level)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+    has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+  }
+
+  template <int kByteWidth>
+  void Roll(const uint8_t* value) {
+    // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+    // true if the hash matches the mask.
+    chunk_size_ += kByteWidth;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (size_t i = 0; i < kByteWidth; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  template <typename T>
+  void Roll(const T* value) {
+    return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+  }
+
+  void Roll(const uint8_t* value, int64_t length) {
+    // Update the rolling hash with a binary-like value, set has_matched_ to true if the
+    // hash matches the mask.
+    chunk_size_ += length;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (auto i = 0; i < length; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  bool NeedNewChunk() {
+    // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+    // set to true if we encountered a match since the last NeedNewChunk() call
+    if (ARROW_PREDICT_FALSE(has_matched_)) {
+      has_matched_ = false;
+      // in order to have a normal distribution of chunk sizes, we only create a new chunk
+      // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+      // different gearhash table (gearhash's chunk size has geometric distribution, and
+      // we use central limit theorem to approximate normal distribution, see
+      // section 6.2.1 in paper https://www.cidrdb.org/cidr2023/papers/p43-low.pdf)
+      if (ARROW_PREDICT_FALSE(++nth_run_ >= kNumGearhashTables)) {
+        // note that we choose not to reset the rolling hash state here, nor anywhere else
+        // in the code, in practice this doesn't seem to affect the chunking effectiveness
+        nth_run_ = 0;
+        chunk_size_ = 0;
+        return true;
+      }
+    }
+    if (ARROW_PREDICT_FALSE(chunk_size_ >= max_chunk_size_)) {
+      // we have a hard limit on the maximum chunk size, note that we don't reset the
+      // rolling hash state here, so the next NeedNewChunk() call will continue from the
+      // current state
+      chunk_size_ = 0;
+      return true;
+    }
+    return false;
+  }
+
+  void ValidateChunks(const std::vector<Chunk>& chunks, int64_t num_levels) const {
+    // chunks must be non-empty and monotonic increasing
+    ARROW_DCHECK(!chunks.empty());
+
+    // the first chunk must start at the first level
+    auto first_chunk = chunks.front();
+    ARROW_DCHECK_EQ(first_chunk.level_offset, 0);
+    ARROW_DCHECK_EQ(first_chunk.value_offset, 0);
+
+    // the following chunks must be contiguous, non-overlapping and monotonically
+    // increasing
+    auto sum_levels = first_chunk.levels_to_write;
+    for (size_t i = 1; i < chunks.size(); ++i) {
+      auto chunk = chunks[i];
+      auto prev_chunk = chunks[i - 1];
+      ARROW_DCHECK_GT(chunk.levels_to_write, 0);
+      ARROW_DCHECK_GE(chunk.value_offset, prev_chunk.value_offset);
+      ARROW_DCHECK_EQ(chunk.level_offset,
+                      prev_chunk.level_offset + prev_chunk.levels_to_write);
+      sum_levels += chunk.levels_to_write;
+    }
+    ARROW_DCHECK_EQ(sum_levels, num_levels);
+
+    // the last chunk must end at the last level
+    auto last_chunk = chunks.back();
+    ARROW_DCHECK_EQ(last_chunk.level_offset + last_chunk.levels_to_write, num_levels);
+  }
+
+  template <typename RollFunc>
+  std::vector<Chunk> Calculate(const int16_t* def_levels, const int16_t* rep_levels,
+                               int64_t num_levels, const RollFunc& RollValue) {
+    // Calculate the chunk boundaries for typed Arrow arrays.

Review Comment:
   The chunking process applies to the entire column across all pages and row groups and only initialized when the column writer is created. This way chunking can be continued between different WriteArrow calls. 
   
   Adding it as comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086152438


##########
python/pyarrow/tests/parquet/test_parquet_writer.py:
##########
@@ -361,3 +361,90 @@ def test_parquet_writer_append_key_value_metadata(tempdir):
     assert metadata[b'key1'] == b'1'
     assert metadata[b'key2'] == b'2'
     assert metadata[b'key3'] == b'3'
+
+
+def test_parquet_content_defined_chunking(tempdir):
+    table = pa.table({'a': range(100_000)})
+
+    # use PLAIN encoding because we compare the overall size of the row groups
+    # which would vary depending on the encoding making the assertions wrong
+    pq.write_table(table, tempdir / 'unchunked.parquet',
+                   use_dictionary=False,
+                   column_encoding="PLAIN")
+    pq.write_table(table, tempdir / 'chunked-default.parquet',
+                   use_dictionary=False,
+                   column_encoding="PLAIN",
+                   use_content_defined_chunking=True)
+    pq.write_table(table, tempdir / 'chunked-custom.parquet',
+                   use_dictionary=False,
+                   column_encoding="PLAIN",
+                   use_content_defined_chunking={"min_chunk_size": 32_768,
+                                                 "max_chunk_size": 65_536})
+
+    # the data must be the same
+    unchunked = pq.read_table(tempdir / 'unchunked.parquet')
+    chunked_default = pq.read_table(tempdir / 'chunked-default.parquet')
+    chunked_custom = pq.read_table(tempdir / 'chunked-custom.parquet')
+    assert unchunked.equals(chunked_default)
+    assert unchunked.equals(chunked_custom)
+
+    # number of row groups and their sizes are not affected by content defined chunking
+    unchunked_metadata = pq.read_metadata(tempdir / 'unchunked.parquet')
+    chunked_default_metadata = pq.read_metadata(tempdir / 'chunked-default.parquet')
+    chunked_custom_metadata = pq.read_metadata(tempdir / 'chunked-custom.parquet')
+
+    assert unchunked_metadata.num_row_groups == chunked_default_metadata.num_row_groups
+    assert unchunked_metadata.num_row_groups == chunked_custom_metadata.num_row_groups
+
+    for i in range(unchunked_metadata.num_row_groups):
+        rg_unchunked = unchunked_metadata.row_group(i)
+        rg_chunked_default = chunked_default_metadata.row_group(i)
+        rg_chunked_custom = chunked_custom_metadata.row_group(i)
+        assert rg_unchunked.num_rows == rg_chunked_default.num_rows
+        assert rg_unchunked.num_rows == rg_chunked_custom.num_rows
+        # since PageReader is not exposed we cannot inspect the page sizes
+        # so just check that the total byte size is different
+        assert rg_unchunked.total_byte_size < rg_chunked_default.total_byte_size
+        assert rg_unchunked.total_byte_size < rg_chunked_custom.total_byte_size
+        assert rg_chunked_default.total_byte_size < rg_chunked_custom.total_byte_size
+
+
+def test_parquet_content_defined_chunking_parameters(tempdir):
+    table = pa.table({'a': range(100)})
+    path = tempdir / 'chunked-invalid.parquet'
+
+    # it raises OSError, not ideal but this is how parquet exceptions are handled
+    # currently

Review Comment:
   Yeah, we should probably have a hierarchy of Parquet exceptions at some point.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086152602


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,412 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+namespace {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_level parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_level Normalization level (default 0)
+// @return The mask used to compare against the rolling hash
+uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size, int norm_level) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be non-negative");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes; also divide by the number of gearhash tables to have a
+  // a more gaussian-like distribution
+  int64_t target_size = (avg_chunk_size - min_chunk_size) / kNumGearhashTables;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // a user defined `norm_level` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_level we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_level is 0 by
+  // default
+  int effective_bits = mask_bits - norm_level;
+
+  if (effective_bits < 1 || effective_bits > 63) {
+    throw ParquetException(
+        "The number of bits in the CDC mask must be between 1 and 63, got " +
+        std::to_string(effective_bits));
+  } else {
+    // create the mask by setting the top bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+}  // namespace
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int norm_level)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_level)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the

Review Comment:
   Actually we should save additional settings to the metadata, like row group size and max page size, otherwise another writer instance could reproduce the same parquet layout. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["wgtmac (via GitHub)" <gi...@apache.org>]

wgtmac commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086194655


##########
cpp/src/parquet/properties.h:
##########
@@ -245,6 +245,34 @@ class PARQUET_EXPORT ColumnProperties {
   bool page_index_enabled_;
 };
 
+// EXPERIMENTAL: Options for content-defined chunking.
+struct PARQUET_EXPORT CdcOptions {
+  /// Minimum chunk size in bytes, default is 256 KiB

Review Comment:
   > A chunk is always a data page, right?
   
   IIUC, it may have many pages in a single chunk if the configured data page size is significantly smaller than the min chunk size. If yes, it would be much clearer if we explain the relationship between the chunk size and data page size configurations.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2087012053


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+namespace {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_level parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_level Normalization level (default 0)
+// @return The mask used to compare against the rolling hash
+uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size, int norm_level) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be non-negative");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes; also divide by the number of gearhash tables to have a
+  // a more gaussian-like distribution
+  int64_t target_size = (avg_chunk_size - min_chunk_size) / kNumGearhashTables;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // a user defined `norm_level` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_level we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_level is 0 by
+  // default
+  int effective_bits = mask_bits - norm_level;
+
+  if (effective_bits < 1 || effective_bits > 63) {
+    throw ParquetException(
+        "The number of bits in the CDC mask must be between 1 and 63, got " +
+        std::to_string(effective_bits));
+  } else {
+    // create the mask by setting the top bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+}  // namespace
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int norm_level)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_level)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+    has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+  }
+
+  template <int kByteWidth>
+  void Roll(const uint8_t* value) {
+    // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+    // true if the hash matches the mask.
+    chunk_size_ += kByteWidth;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (size_t i = 0; i < kByteWidth; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  template <typename T>
+  void Roll(const T* value) {
+    return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+  }
+
+  void Roll(const uint8_t* value, int64_t length) {
+    // Update the rolling hash with a binary-like value, set has_matched_ to true if the
+    // hash matches the mask.
+    chunk_size_ += length;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (auto i = 0; i < length; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  bool NeedNewChunk() {
+    // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+    // set to true if we encountered a match since the last NeedNewChunk() call
+    if (ARROW_PREDICT_FALSE(has_matched_)) {
+      has_matched_ = false;
+      // in order to have a normal distribution of chunk sizes, we only create a new chunk
+      // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+      // different gearhash table (gearhash's chunk size has geometric distribution, and
+      // we use central limit theorem to approximate normal distribution, see
+      // section 6.2.1 in paper https://www.cidrdb.org/cidr2023/papers/p43-low.pdf)
+      if (ARROW_PREDICT_FALSE(++nth_run_ >= kNumGearhashTables)) {
+        // note that we choose not to reset the rolling hash state here, nor anywhere else
+        // in the code, in practice this doesn't seem to affect the chunking effectiveness
+        nth_run_ = 0;
+        chunk_size_ = 0;
+        return true;
+      }
+    }
+    if (ARROW_PREDICT_FALSE(chunk_size_ >= max_chunk_size_)) {
+      // we have a hard limit on the maximum chunk size, note that we don't reset the
+      // rolling hash state here, so the next NeedNewChunk() call will continue from the
+      // current state
+      chunk_size_ = 0;
+      return true;
+    }
+    return false;
+  }
+
+  void ValidateChunks(const std::vector<Chunk>& chunks, int64_t num_levels) const {
+    // chunks must be non-empty and monotonic increasing
+    ARROW_DCHECK(!chunks.empty());
+
+    // the first chunk must start at the first level
+    auto first_chunk = chunks.front();
+    ARROW_DCHECK_EQ(first_chunk.level_offset, 0);
+    ARROW_DCHECK_EQ(first_chunk.value_offset, 0);
+
+    // the following chunks must be contiguous, non-overlapping and monotonically
+    // increasing
+    auto sum_levels = first_chunk.levels_to_write;
+    for (size_t i = 1; i < chunks.size(); ++i) {
+      auto chunk = chunks[i];
+      auto prev_chunk = chunks[i - 1];
+      ARROW_DCHECK_GT(chunk.levels_to_write, 0);
+      ARROW_DCHECK_GE(chunk.value_offset, prev_chunk.value_offset);
+      ARROW_DCHECK_EQ(chunk.level_offset,
+                      prev_chunk.level_offset + prev_chunk.levels_to_write);
+      sum_levels += chunk.levels_to_write;
+    }
+    ARROW_DCHECK_EQ(sum_levels, num_levels);
+
+    // the last chunk must end at the last level
+    auto last_chunk = chunks.back();
+    ARROW_DCHECK_EQ(last_chunk.level_offset + last_chunk.levels_to_write, num_levels);
+  }
+
+  template <typename RollFunc>
+  std::vector<Chunk> Calculate(const int16_t* def_levels, const int16_t* rep_levels,
+                               int64_t num_levels, const RollFunc& RollValue) {
+    // Calculate the chunk boundaries for typed Arrow arrays.

Review Comment:
   Added a more detailed description to the method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1946143584


##########
cpp/src/parquet/column_chunker.h:
##########
@@ -0,0 +1,784 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+using arrow::internal::checked_cast;
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {

Review Comment:
   Need to find a better way to embed these constants, I used the following script to generate this:
   
   ```py
   import hashlib
   import sys
   
   
   def gear(n: int, seed: int):
       value = bytes([seed] * 64 + [n] * 64)
       hasher = hashlib.md5(value)
       return hasher.hexdigest()[:16]
   
   
   def print_table(seed: int, length=256, comma=True):
       table = [gear(n, seed=seed) for n in range(length)]
       print(f"{{ // seed = {seed}")
       for i in range(0, length, 4):
           print("  ", end="")
           values = [f"0x{value}" for value in table[i:i + 4]]
           values = ", ".join(values)
           print(f"  {values}", end=",\n" if i < length - 4 else "\n")
       print(" }", end=", " if comma else "")
   
   
   
   if __name__ == "__main__":
       print("{")
       n = int(sys.argv[1])
       for seed in range(n):
           print_table(seed, comma=seed < n)
       print("}")
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1981154673


##########
cpp/src/parquet/column_chunker_hashtable.py:
##########
@@ -0,0 +1,90 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import hashlib
+import pathlib
+import sys
+from io import StringIO
+
+
+template = """\
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+#include <cstdint>
+
+namespace parquet {{
+namespace internal {{

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1981297868


##########
cpp/src/parquet/column_chunker.h:
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+
+namespace internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage systems (CAS)
+/// which split the bytes stream into content defined blobs. The CAS system will calculate
+/// a unique identifier for each blob, then store the blob in a key-value store. If the
+/// same blob is encountered again, the system can refer to the hash instead of physically
+/// storing the blob again. In the example above, the CAS system would phiysically store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks///
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC paper: "FastCDC: a Fast and Efficient Content-Defined Chunking Approach for
+/// Data Deduplication"
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+class ContentDefinedChunker {
+ public:
+  /// Create a new ContentDefinedChunker instance
+  ///
+  /// @param level_info Information about definition and repetition levels
+  /// @param size_range Min/max chunk size as pair<min_size, max_size>, the chunker will
+  ///                   attempt to uniformly distribute the chunks between these extremes.
+  /// @param norm_factor Normalization factor to center the chunk size around the average
+  ///                    size more aggressively. By increasing the normalization factor,
+  ///                    probability of finding a chunk boundary increases.
+  ContentDefinedChunker(const LevelInfo& level_info,
+                        std::pair<uint64_t, uint64_t> size_range,
+                        uint8_t norm_factor = 0);
+
+  /// Get the chunk boundaries for the given column data
+  ///
+  /// @param def_levels Definition levels
+  /// @param rep_levels Repetition levels
+  /// @param num_levels Number of levels
+  /// @param values Column values as an Arrow array
+  /// @return Vector of Chunk objects representing the chunk boundaries
+  const ::arrow::Result<std::vector<Chunk>> GetBoundaries(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const ::arrow::Array& values);

Review Comment:
   Yes, CDC is not supported by `WriteBatch` and `WriteBatchSpaced` at the moment because `WriteArrow` dispatches through those in certain cases. I can see two ways to overcome that:
   1. Rename `WriteBatch` and `WriteBatchSpaced` to `WriteBatchInternal` and `WriteBatchSpacedInternal` respectively and use the same CDC chunking logic in the exposed `WriteBatch` and `WriteBatchSpaced` public methods. This is rather straightforward and we would provide the same functionality regardless of the used API.
   2. Generalize the chunker functionality and consolidate the offset calculation logic spread across `DoInBatches`, `WriteLevels`, `WriteLevelsSpaced` and `MaybeCalculateValidityBits`. This would be a more invasive change and while I would like to tackle it, ideally this should be deferred to a follow-up PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1981277569


##########
cpp/src/parquet/column_chunker.h:
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+
+namespace internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage systems (CAS)
+/// which split the bytes stream into content defined blobs. The CAS system will calculate
+/// a unique identifier for each blob, then store the blob in a key-value store. If the
+/// same blob is encountered again, the system can refer to the hash instead of physically
+/// storing the blob again. In the example above, the CAS system would phiysically store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks///
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC paper: "FastCDC: a Fast and Efficient Content-Defined Chunking Approach for
+/// Data Deduplication"
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+class ContentDefinedChunker {

Review Comment:
   Renamed all relevant files to `chunker_internal_*`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1985312393


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, uint64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal128Builder, ::arrow::Decimal128(val % max_value))
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal256Builder, ::arrow::Decimal256(val % max_value))
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      GENERATE_CASE_BODY(::arrow::FixedSizeBinaryBuilder,
+                         std::string("bin_") + std::to_string(val).substr(0, size - 4))
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, uint64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(const std::shared_ptr<Table>& table,
+                                                   uint64_t min_chunk_size,
+                                                   uint64_t max_chunk_size,
+                                                   bool enable_dictionary = false,
+
+                                                   int64_t row_group_size = 1024 * 1024) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_cdc()
+      ->cdc_size_range(min_chunk_size, max_chunk_size)
+      ->cdc_norm_factor(0);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+struct PageSizes {
+  std::vector<uint64_t> lengths;
+  std::vector<uint64_t> sizes;
+};
+
+PageSizes GetColumnPageSizes(const std::shared_ptr<Buffer>& data, int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  PageSizes result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        result.sizes.push_back(data_page->size());
+        result.lengths.push_back(data_page->num_values());
+      }
+    }
+  }
+
+  return result;
+}
+
+Result<PageSizes> WriteAndGetPageSizes(const std::shared_ptr<Table>& table,
+                                       uint64_t min_chunk_size, uint64_t max_chunk_size,
+                                       bool enable_dictionary = false,
+                                       int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnPageSizes(buffer, column_index);
+}
+
+void AssertAllBetween(const std::vector<uint64_t>& values, uint64_t min, uint64_t max,
+                      bool expect_dictionary_fallback = false) {
+  // expect the last chunk since it is not guaranteed to be within the range
+  if (expect_dictionary_fallback) {
+    // if dictionary encoding is enabled, the writer can fallback to plain
+    // encoding splitting within a content defined chunk, so we can't
+    // guarantee that all chunks are within the range in this case, but we
+    // know that there can be at most 2 pages smaller than the min_chunk_size
+    size_t smaller_count = 0;
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      if (values[i] < min) {
+        smaller_count++;
+      } else {
+        ASSERT_LE(values[i], max);
+      }
+    }
+    ASSERT_LE(smaller_count, 2);
+  } else {
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      ASSERT_GE(values[i], min);
+      ASSERT_LE(values[i], max);
+    }
+  }
+  ASSERT_LE(values.back(), max);
+}
+
+std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>> FindDifferences(

Review Comment:
   Updated, added ChunkList, ChunkDiff types for better readability.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1983869547


##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}

Review Comment:
   Well, required for a more convenient `.emplate_back()`:
   
   ```
   /Users/kszucs/Workspace/arrow/cpp/src/parquet/chunker_internal.cc:183:18: note: in instantiation of function template specialization 'std::vector<parquet::internal::Chunk>::emplace_back<long long &, long long &, long long &>' requested here
     183 |           chunks.emplace_back(record_level_offset, record_value_offset, levels_to_write);
     ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1985541624


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, uint64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal128Builder, ::arrow::Decimal128(val % max_value))
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal256Builder, ::arrow::Decimal256(val % max_value))
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      GENERATE_CASE_BODY(::arrow::FixedSizeBinaryBuilder,
+                         std::string("bin_") + std::to_string(val).substr(0, size - 4))
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, uint64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(const std::shared_ptr<Table>& table,
+                                                   uint64_t min_chunk_size,
+                                                   uint64_t max_chunk_size,
+                                                   bool enable_dictionary = false,
+
+                                                   int64_t row_group_size = 1024 * 1024) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_cdc()
+      ->cdc_size_range(min_chunk_size, max_chunk_size)
+      ->cdc_norm_factor(0);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+struct PageSizes {
+  std::vector<uint64_t> lengths;
+  std::vector<uint64_t> sizes;
+};
+
+PageSizes GetColumnPageSizes(const std::shared_ptr<Buffer>& data, int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  PageSizes result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        result.sizes.push_back(data_page->size());
+        result.lengths.push_back(data_page->num_values());
+      }
+    }
+  }
+
+  return result;
+}
+
+Result<PageSizes> WriteAndGetPageSizes(const std::shared_ptr<Table>& table,
+                                       uint64_t min_chunk_size, uint64_t max_chunk_size,
+                                       bool enable_dictionary = false,
+                                       int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnPageSizes(buffer, column_index);
+}
+
+void AssertAllBetween(const std::vector<uint64_t>& values, uint64_t min, uint64_t max,
+                      bool expect_dictionary_fallback = false) {
+  // expect the last chunk since it is not guaranteed to be within the range
+  if (expect_dictionary_fallback) {
+    // if dictionary encoding is enabled, the writer can fallback to plain
+    // encoding splitting within a content defined chunk, so we can't
+    // guarantee that all chunks are within the range in this case, but we
+    // know that there can be at most 2 pages smaller than the min_chunk_size
+    size_t smaller_count = 0;
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      if (values[i] < min) {
+        smaller_count++;
+      } else {
+        ASSERT_LE(values[i], max);
+      }
+    }
+    ASSERT_LE(smaller_count, 2);
+  } else {
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      ASSERT_GE(values[i], min);
+      ASSERT_LE(values[i], max);
+    }
+  }
+  ASSERT_LE(values.back(), max);
+}
+
+std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>> FindDifferences(
+    const std::vector<uint64_t>& first, const std::vector<uint64_t>& second) {
+  // Compute LCS table.

Review Comment:
   Added more description to the utility function.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1983600346


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {

Review Comment:
   With `norm_factor=8`
   
   ![newplot-14](https://github.com/user-attachments/assets/0593afaf-502d-4694-be63-b72f249a26de)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1988119205


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,250 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_hashtable.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size

Review Comment:
   Yes, it is pretty close by default using `norm_factor=0`, `min_size=256KiB`, `max_size=1MiB` 
   
   ![norm-factor-0](https://github.com/user-attachments/assets/80962907-c30d-4de9-98fb-45e7afd51fb5)
   
   Using `norm_factor=1`:
   
   ![norm-factor-1](https://github.com/user-attachments/assets/136f6d47-9544-40aa-b893-2acc62d61476)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1968426044


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {

Review Comment:
   We have 8 independent gear hash tables in order to normalize chunk sizes, see more details [in the comments here](https://github.com/apache/arrow/pull/45360/files#diff-b0e0dc9efa0271723fc8f200c7a54465dd861655788434ee2884d9bac72e83c2R622)
   
   It helps to achieve chunk size distributions like the following:
   ![newplot-8](https://github.com/user-attachments/assets/9b3224df-e495-4d05-8307-4ca930096da1)
   61655788434ee2884d9bac72e83c2R622).
   
   The spikes on the left are because of the dictionary encoding and the right because of the max chunk size limitation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1968426044


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {

Review Comment:
   We have 8 independent gear hash tables in order to normalize chunk sizes, see more details in the comments at https://github.com/apache/arrow/pull/45360/files#diff-b0e0dc9efa0271723fc8f200c7a54465dd861655788434ee2884d9bac72e83c2R622



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1992222289


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size

Review Comment:
   Correct. I could adjust that by increasing the min_size to stay centered around the avg_size. 
   I am going to try this out and run a couple of examples.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1970562371


##########
cpp/src/parquet/column_chunker_test.cc:
##########
@@ -0,0 +1,891 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+// using ::arrow::random::GenerateBatch;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,

Review Comment:
   I hand crafted a generator to have consistent data across platforms as well. Ideally this could be done using the existing random array generator machinery but with a custom callable rather than `pcg32_fast`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1974002195


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for

Review Comment:
   I think we can stick with gearhash.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1974006613


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {

Review Comment:
   Sure, what would be the right place for this? Just in the same directory?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1973217654


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+bool ContentDefinedChunker::Check() {

Review Comment:
   Definitely!



##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+bool ContentDefinedChunker::Check() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last Check() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling
+    // hash state here, so the next Check() call will continue from the current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename T>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const T& leaf_array) {
+  std::vector<Chunk> result;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(def_levels[offset]);
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else {
+    // nested data with nulls
+    bool has_leaf_value;
+    bool is_record_boundary;
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t level_offset = 0;
+    int64_t value_offset = 0;
+    int64_t record_level_offset = 0;
+    int64_t record_value_offset = 0;
+
+    while (level_offset < num_levels) {

Review Comment:
   Updating.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1973241927


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }

Review Comment:
   It is in the critical path, I am going to try this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2028665323


##########
python/pyarrow/tests/parquet/test_parquet_writer.py:
##########
@@ -361,3 +361,82 @@ def test_parquet_writer_append_key_value_metadata(tempdir):
     assert metadata[b'key1'] == b'1'
     assert metadata[b'key2'] == b'2'
     assert metadata[b'key3'] == b'3'
+
+
+def test_parquet_content_defined_chunking(tempdir):
+    table = pa.table({'a': range(100_000)})
+
+    pq.write_table(table, tempdir / 'unchunked.parquet')
+    pq.write_table(table, tempdir / 'chunked-default.parquet',
+                   use_content_defined_chunking=True)
+    pq.write_table(table, tempdir / 'chunked-custom.parquet',
+                   use_content_defined_chunking={"min_chunk_size": 32_768,
+                                                 "max_chunk_size": 65_536})
+
+    # the data must be the same
+    unchunked = pq.read_table(tempdir / 'unchunked.parquet')
+    chunked_default = pq.read_table(tempdir / 'chunked-default.parquet')
+    chunked_custom = pq.read_table(tempdir / 'chunked-custom.parquet')
+    assert unchunked.equals(chunked_default)
+    assert unchunked.equals(chunked_custom)
+
+    # number of row groups and their sizes are not affected by content defined chunking
+    unchunked_metadata = pq.read_metadata(tempdir / 'unchunked.parquet')
+    chunked_default_metadata = pq.read_metadata(tempdir / 'chunked-default.parquet')
+    chunked_custom_metadata = pq.read_metadata(tempdir / 'chunked-custom.parquet')
+
+    assert unchunked_metadata.num_row_groups == chunked_default_metadata.num_row_groups
+    assert unchunked_metadata.num_row_groups == chunked_custom_metadata.num_row_groups
+
+    for i in range(unchunked_metadata.num_row_groups):
+        rg_unchunked = unchunked_metadata.row_group(i)
+        rg_chunked_default = chunked_default_metadata.row_group(i)
+        rg_chunked_custom = chunked_custom_metadata.row_group(i)
+        assert rg_unchunked.num_rows == rg_chunked_default.num_rows
+        assert rg_unchunked.num_rows == rg_chunked_custom.num_rows
+        # since PageReader is not exposed we don't cannot inspect the page sizes
+        # so just check that the total byte size is different

Review Comment:
   Interestingly this is only true if we use PLAIN encoding. With the default encoding the chunked parquet files were actually smaller for this data sample but turned over for bigger samples.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025230307


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int8_t norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes
+  int64_t target_size = avg_chunk_size - min_chunk_size;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // 3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  if (norm_factor < -3 || norm_factor > 3) {
+    ARROW_LOG(WARNING) << "norm_factor=" << norm_factor
+                       << " is outside the recommended range [-3, 3]";
+  }
+
+  int mask_adjustement = 3 + norm_factor;
+  int effective_bits = mask_bits - mask_adjustement;
+  if (effective_bits == 0) {
+    return 0;
+  } else if (effective_bits > 64) {
+    throw ParquetException("The number of bits in the mask cannot exceed 64, got " +
+                           std::to_string(effective_bits));
+  } else if (effective_bits < 0) {
+    throw ParquetException(
+        "The difference between min_chunk_size=" + std::to_string(min_chunk_size) +
+        " and max_chunk_size=" + std::to_string(max_chunk_size) +
+        " is too small for the given norm_factor=" + std::to_string(norm_factor) +
+        ", either increase the size range or decrease the norm_factor.");
+  } else {
+    // create the mask by setting the top mask_bits bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int8_t norm_factor)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_factor)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(const bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+    has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+  }
+
+  template <int ByteWidth>
+  void Roll(const uint8_t* value) {
+    // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+    // true if the hash matches the mask.
+
+    chunk_size_ += ByteWidth;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (size_t i = 0; i < ByteWidth; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  template <typename T>
+  void Roll(const T* value) {
+    return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+  }
+
+  void Roll(const uint8_t* value, int64_t length) {
+    // Update the rolling hash with a binary-like value, set has_matched_ to true if the
+    // hash matches the mask.
+
+    chunk_size_ += length;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (auto i = 0; i < length; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  bool NeedNewChunk() {
+    // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+    // set to true if we encountered a match since the last NeedNewChunk() call
+    if (ARROW_PREDICT_FALSE(has_matched_)) {
+      has_matched_ = false;
+      // in order to have a normal distribution of chunk sizes, we only create a new chunk
+      // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+      // different gearhash table (gearhash's chunk size has geometric distribution, and
+      // we use central limit theorem to approximate normal distribution, see
+      // section 6.2.1 in paper https://www.cidrdb.org/cidr2023/papers/p43-low.pdf)
+      if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+        nth_run_ = 0;
+        chunk_size_ = 0;
+        return true;
+      }
+    }
+    if (ARROW_PREDICT_FALSE(chunk_size_ >= max_chunk_size_)) {
+      // we have a hard limit on the maximum chunk size, note that we don't reset the
+      // rolling hash state here, so the next NeedNewChunk() call will continue from the
+      // current state
+      chunk_size_ = 0;
+      return true;
+    }
+    return false;
+  }
+
+  void ValidateChunks(const std::vector<Chunk>& chunks, int64_t num_levels) {

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025710442


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();
+    auto min_length = ElementCount(min_chunk_size, byte_width, nullable);
+    auto max_length = ElementCount(max_chunk_size, byte_width, nullable);
+    AssertAllBetween(base_info.page_lengths, min_length, max_length);
+    AssertAllBetween(modified_info.page_lengths, min_length, max_length);
+  } else if (::arrow::is_base_binary_like(dtype->id()) && !enable_dictionary) {
+    AssertAllBetween(base_info.page_sizes, min_chunk_size, max_chunk_size);
+    AssertAllBetween(modified_info.page_sizes, min_chunk_size, max_chunk_size);
+  }
+}
+
+constexpr int64_t kMinChunkSize = 8 * 1024;
+constexpr int64_t kMaxChunkSize = 32 * 1024;
+constexpr int64_t kPartSize = 128 * 1024;
+constexpr int64_t kEditSize = 128;
+
+struct CaseConfig {
+  // Arrow data type to generate the testing data for
+  std::shared_ptr<::arrow::DataType> dtype;
+  // Whether the data type is nullable
+  bool is_nullable;
+  // Approximate number of bytes per record to calculate the number of elements to
+  // generate
+  size_t bytes_per_record;
+  // Data page version to use
+  ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1;
+};
+
+// Define PrintTo for MyStruct
+void PrintTo(const CaseConfig& param, std::ostream* os) {
+  *os << "{ " << param.dtype->ToString();
+  if (param.is_nullable) {
+    *os << " nullable";
+  }
+  *os << " }";
+}
+
+class TestCDC : public ::testing::Test {
+ public:
+  uint64_t GetMask(const ContentDefinedChunker& cdc) const { return cdc.GetMask(); }
+};
+
+TEST_F(TestCDC, RollingHashMaskCalculation) {
+  auto le = LevelInfo();
+  auto min_size = 256 * 1024;
+  auto max_size = 1024 * 1024;
+
+  auto cdc0 = ContentDefinedChunker(le, min_size, max_size, 0);
+  ASSERT_EQ(GetMask(cdc0), 0xFFFE000000000000);
+
+  auto cdc1 = ContentDefinedChunker(le, min_size, max_size, 1);
+  ASSERT_EQ(GetMask(cdc1), 0xFFFC000000000000);
+
+  auto cdc2 = ContentDefinedChunker(le, min_size, max_size, 2);
+  ASSERT_EQ(GetMask(cdc2), 0xFFF8000000000000);
+
+  auto cdc3 = ContentDefinedChunker(le, min_size, max_size, 3);
+  ASSERT_EQ(GetMask(cdc3), 0xFFF0000000000000);
+
+  auto cdc4 = ContentDefinedChunker(le, min_size, max_size, -1);
+  ASSERT_EQ(GetMask(cdc4), 0xFFFF000000000000);
+
+  // this is the smallest possible mask always matching, by using 8 hashtables
+  // we are going to have a match every 8 bytes; this is an unrealistic case
+  // but checking for the correctness of the mask calculation
+  auto cdc5 = ContentDefinedChunker(le, 0, 16, 0);
+  ASSERT_EQ(GetMask(cdc5), 0x0000000000000000);
+
+  auto cdc6 = ContentDefinedChunker(le, 0, 32, 1);
+  ASSERT_EQ(GetMask(cdc6), 0x0000000000000000);
+
+  auto cdc7 = ContentDefinedChunker(le, 0, 16, -1);
+  ASSERT_EQ(GetMask(cdc7), 0x8000000000000000);
+
+  // another unrealistic case, checking for the validation
+  auto cdc8 = ContentDefinedChunker(le, 128, 384, -60);
+  ASSERT_EQ(GetMask(cdc8), 0xFFFFFFFFFFFFFFFF);
+}
+
+TEST_F(TestCDC, WriteSingleColumnParquetFile) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  std::vector<int32_t> numbers = {1, 2, 3, 4, 5};
+  std::vector<uint8_t> valid_bits = {1, 0, 1, 0, 1};
+  EXPECT_THROW(
+      int_column_writer.WriteBatch(numbers.size(), nullptr, nullptr, numbers.data()),
+      ParquetException);
+  EXPECT_THROW(int_column_writer.WriteBatchSpaced(numbers.size(), nullptr, nullptr,
+                                                  valid_bits.data(), 0, numbers.data()),
+               ParquetException);
+}
+
+TEST_F(TestCDC, LastChunkDoesntTriggerAddDataPage) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()
+                   ->content_defined_chunking_options(kMinChunkSize, kMaxChunkSize, 0)
+                   ->disable_dictionary()
+                   ->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  ASSERT_OK_AND_ASSIGN(auto array, ::arrow::gen::Step()->Generate(8000));
+  auto arrow_props = default_arrow_writer_properties();
+  auto arrow_ctx = ArrowWriteContext(default_memory_pool(), arrow_props.get());
+
+  // Calling WriteArrow twice, we expect that the first call doesn't add a new data page
+  // at the end allowing subsequent calls to append to the same page
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+
+  int_column_writer.Close();
+  writer->Close();
+  ASSERT_OK_AND_ASSIGN(auto buffer, sink->Finish());
+
+  auto info = GetColumnParquetInfo(buffer);
+  ASSERT_EQ(info.size(), 1);
+
+  // AssertAllBetween allow the last chunk size to be smaller than the min_chunk_size
+  AssertAllBetween(info[0].page_sizes, kMinChunkSize, kMaxChunkSize);
+  AssertAllBetween(info[0].page_lengths, 3000, 5000);
+}
+
+TEST_F(TestCDC, ChunkSizeParameterValidation) {
+  // Test that constructor validates min/max chunk size parameters
+  auto li = LevelInfo();
+
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 256 * 1024, 1024 * 1024));
+
+  // with norm_factor=0 the difference between min and max chunk size must be
+  // at least 16
+  ASSERT_THROW(ContentDefinedChunker(li, 0, -1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1024, 512), ParquetException);
+
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16));
+  ASSERT_THROW(ContentDefinedChunker(li, -16, -16), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 16, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 32, 32), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 32, 48));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1024 * 1024, 2 * 1024 * 1024));
+  ASSERT_NO_THROW(
+      ContentDefinedChunker(li, 1024 * 1024 * 1024L, 2LL * 1024 * 1024 * 1024L));
+
+  // with norm_factor=1 the difference between min and max chunk size must be
+  // at least 64
+  ASSERT_THROW(ContentDefinedChunker(li, 1, -1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 32, 1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1, 33, 1));
+
+  // with norm_factor=2 the difference between min and max chunk size must be
+  // at least 128
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 63, 2), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 64, 2));
+
+  // with norm_factor=-1 the difference between min and max chunk size must be
+  // at least 8
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 7, -1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 8, -1));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16, -1));
+
+  // test the norm_factor extremes
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0, -68), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 1, -67));
+  ASSERT_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 59),
+               ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 58));
+}
+
+class TestCDCSingleRowGroup : public ::testing::TestWithParam<CaseConfig> {
+ protected:
+  // Column random table parts for testing
+  std::shared_ptr<Field> field_;
+  std::shared_ptr<Table> part1_, part2_, part3_, part4_, part5_, part6_, part7_;
+
+  void SetUp() override {
+    const auto& param = GetParam();
+    auto field_ = ::arrow::field("f0", param.dtype, param.is_nullable);
+    auto schema = ::arrow::schema({field_});
+
+    auto part_length = kPartSize / param.bytes_per_record;
+    auto edit_length = kEditSize / param.bytes_per_record;
+    ASSERT_OK_AND_ASSIGN(part1_, GenerateTable(schema, part_length, 0));
+    ASSERT_OK_AND_ASSIGN(part2_, GenerateTable(schema, edit_length, 1));
+    ASSERT_OK_AND_ASSIGN(part3_, GenerateTable(schema, part_length, part_length));
+    ASSERT_OK_AND_ASSIGN(part4_, GenerateTable(schema, edit_length, 2));
+    ASSERT_OK_AND_ASSIGN(part5_, GenerateTable(schema, part_length, 2 * part_length));
+    ASSERT_OK_AND_ASSIGN(part6_, GenerateTable(schema, edit_length, 3));
+    ASSERT_OK_AND_ASSIGN(part7_, GenerateTable(schema, edit_length, 4));
+  }
+};
+
+TEST_P(TestCDCSingleRowGroup, DeleteOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/0,
+                                /*exact_number_of_smaller_diffs=*/1, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, DeleteTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part3_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/0,
+                                /*exact_number_of_smaller_diffs=*/2, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, UpdateOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part4_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*max_number_of_equal_diffs=*/1);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, UpdateTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified,
+                       ConcatAndCombine({part1_, part6_, part3_, part7_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*max_number_of_equal_diffs=*/2);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, InsertOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/1,
+                                /*exact_number_of_smaller_diffs=*/0, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, InsertTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part3_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/2,
+                                /*exact_number_of_smaller_diffs=*/0, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, Append) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part2_, part3_, part4_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto original_page_lengths = base_info.front().page_lengths;
+    auto modified_page_lengths = modified_info.front().page_lengths;
+    ASSERT_GE(original_page_lengths.size(), modified_page_lengths.size());
+    for (size_t i = 0; i < original_page_lengths.size() - 1; i++) {
+      ASSERT_EQ(original_page_lengths[i], modified_page_lengths[i]);
+    }
+    ASSERT_GT(modified_page_lengths.back(), original_page_lengths.back());
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, EmptyTable) {
+  const auto& param = GetParam();
+
+  auto schema = ::arrow::schema({::arrow::field("f0", param.dtype, param.is_nullable)});
+  ASSERT_OK_AND_ASSIGN(auto empty_table, GenerateTable(schema, 0, 0));
+  ASSERT_EQ(empty_table->num_rows(), 0);
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(
+        auto result, WriteAndGetParquetInfo(empty_table, kMinChunkSize, kMaxChunkSize,
+                                            enable_dictionary, param.data_page_version));
+
+    // An empty table should result in no data pages
+    ASSERT_EQ(result.size(), 1);
+    ASSERT_TRUE(result.front().page_lengths.empty());
+    ASSERT_TRUE(result.front().page_sizes.empty());
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, ArrayOffsets) {
+  const auto& param = GetParam();
+  ASSERT_OK_AND_ASSIGN(auto table, ConcatAndCombine({part1_, part2_, part3_}));
+
+  for (auto offset : {0, 512, 1024}) {
+    auto sliced_table = table->Slice(offset);
+
+    // assert that the first column has a non-zero offset
+    auto column = sliced_table->column(0);
+    auto first_chunk = column->chunk(0);
+    ASSERT_EQ(first_chunk->offset(), offset);
+
+    // write out the sliced table, read it back and compare
+    ASSERT_OK(WriteAndGetParquetInfo(sliced_table, kMinChunkSize, kMaxChunkSize, true,
+                                     param.data_page_version));
+  }
+}
+
+INSTANTIATE_TEST_SUITE_P(
+    FixedSizedTypes, TestCDCSingleRowGroup,
+    testing::Values(
+        // Boolean
+        CaseConfig{::arrow::boolean(), false, 1},
+        // Numeric
+        CaseConfig{::arrow::uint8(), false, 1}, CaseConfig{::arrow::uint16(), false, 2},
+        CaseConfig{::arrow::uint32(), false, 4}, CaseConfig{::arrow::uint64(), true, 8},
+        CaseConfig{::arrow::int8(), false, 1}, CaseConfig{::arrow::int16(), false, 2},
+        CaseConfig{::arrow::int32(), false, 4}, CaseConfig{::arrow::int64(), true, 8},
+        CaseConfig{::arrow::float16(), false, 2},
+        CaseConfig{::arrow::float32(), false, 4}, CaseConfig{::arrow::float64(), true, 8},
+        CaseConfig{::arrow::decimal128(18, 6), false, 16},
+        CaseConfig{::arrow::decimal256(40, 6), false, 32},
+        // Binary-like
+        CaseConfig{::arrow::utf8(), false, 16}, CaseConfig{::arrow::binary(), true, 16},
+        CaseConfig{::arrow::fixed_size_binary(16), true, 16},
+        // Temporal
+        CaseConfig{::arrow::date32(), false, 4},
+        CaseConfig{::arrow::time32(::arrow::TimeUnit::MILLI), true, 4},
+        CaseConfig{::arrow::time64(::arrow::TimeUnit::NANO), false, 8},
+        CaseConfig{::arrow::timestamp(::arrow::TimeUnit::NANO), true, 8},
+        CaseConfig{::arrow::duration(::arrow::TimeUnit::NANO), false, 8},
+        // Nested types
+        CaseConfig{::arrow::list(::arrow::int32()), false, 16},
+        CaseConfig{::arrow::list(::arrow::int32()), true, 18},
+        CaseConfig{::arrow::list(::arrow::utf8()), true, 18},
+        CaseConfig{::arrow::struct_({::arrow::field("f0", ::arrow::int32())}), false, 8},
+        CaseConfig{::arrow::struct_({::arrow::field("f0", ::arrow::float64())}), true,
+                   10},
+        CaseConfig{
+            ::arrow::list(::arrow::struct_({::arrow::field("f0", ::arrow::int32())})),
+            false, 16},
+        // Extension type
+        CaseConfig{::arrow::uuid(), true, 16},
+        // Use ParquetDataPageVersion::V2
+        CaseConfig{::arrow::large_binary(), false, 16, ParquetDataPageVersion::V2},
+        CaseConfig{::arrow::list(::arrow::utf8()), true, 18,
+                   ParquetDataPageVersion::V2}));
+
+class TestCDCMultipleRowGroups : public ::testing::Test {
+ protected:
+  // Column random table parts for testing
+  std::shared_ptr<DataType> dtype_;
+  std::shared_ptr<Table> part1_, part2_, part3_;
+  std::shared_ptr<Table> edit1_, edit2_, edit3_;
+
+  void SetUp() override {
+    auto constexpr kPartLength = 256 * 1024;
+    auto constexpr kEditLength = 128;
+
+    dtype_ = ::arrow::int32();
+    auto field = ::arrow::field("f0", dtype_, true);
+    auto schema = ::arrow::schema({field});
+
+    ASSERT_OK_AND_ASSIGN(part1_, GenerateTable(schema, kPartLength, 0));
+    ASSERT_OK_AND_ASSIGN(part2_, GenerateTable(schema, kPartLength, 2));
+    ASSERT_OK_AND_ASSIGN(part3_, GenerateTable(schema, kPartLength, 4));
+
+    ASSERT_OK_AND_ASSIGN(edit1_, GenerateTable(schema, kEditLength, 1));
+    ASSERT_OK_AND_ASSIGN(edit2_, GenerateTable(schema, kEditLength, 3));
+    ASSERT_OK_AND_ASSIGN(edit3_, GenerateTable(schema, kEditLength, 5));
+  }
+};
+
+TEST_F(TestCDCMultipleRowGroups, InsertOnce) {
+  auto constexpr kRowGroupLength = 128 * 1024;
+  auto constexpr kEnableDictionary = false;
+  auto constexpr kMinChunkSize = 0 * 1024;
+  auto constexpr kMaxChunkSize = 128 * 1024;
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, edit1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto inserted,
+                       ConcatAndCombine({part1_, edit1_, edit2_, part2_, part3_}));
+  ASSERT_FALSE(base->Equals(*inserted));
+  ASSERT_EQ(inserted->num_rows(), base->num_rows() + edit2_->num_rows());
+
+  ASSERT_OK_AND_ASSIGN(
+      auto base_info,
+      WriteAndGetParquetInfo(base, kMinChunkSize, kMaxChunkSize, kEnableDictionary,
+                             ParquetDataPageVersion::V1, kRowGroupLength));
+  ASSERT_OK_AND_ASSIGN(
+      auto inserted_info,
+      WriteAndGetParquetInfo(inserted, kMinChunkSize, kMaxChunkSize, kEnableDictionary,
+                             ParquetDataPageVersion::V1, kRowGroupLength));
+
+  ASSERT_EQ(base_info.size(), 7);
+  ASSERT_EQ(inserted_info.size(), 7);
+
+  ASSERT_EQ(base_info.at(0).page_lengths, inserted_info.at(0).page_lengths);
+  ASSERT_EQ(base_info.at(1).page_lengths, inserted_info.at(1).page_lengths);
+  for (size_t i = 2; i < inserted_info.size() - 1; i++) {
+    AssertPageLengthDifferences(base_info.at(i), inserted_info.at(i),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/1,
+                                /*exact_number_of_smaller_diffs=*/1, edit2_->num_rows());
+  }
+  AssertPageLengthDifferences(base_info.back(), inserted_info.back(),
+                              /*exact_number_of_equal_diffs=*/0,
+                              /*exact_number_of_larger_diffs=*/1,
+                              /*exact_number_of_smaller_diffs=*/0, edit2_->num_rows());
+}
+
+TEST_F(TestCDCMultipleRowGroups, DeleteOnce) {
+  auto constexpr kRowGroupLength = 128 * 1024;
+  auto constexpr kEnableDictionary = false;
+  auto constexpr kMinChunkSize = 0 * 1024;
+  auto constexpr kMaxChunkSize = 128 * 1024;
+
+  ASSERT_OK_AND_ASSIGN(auto base,
+                       ConcatAndCombine({part1_, edit1_, part2_, part3_, edit2_}));
+  ASSERT_OK_AND_ASSIGN(auto deleted, ConcatAndCombine({part1_, part2_, part3_, edit2_}));
+  ASSERT_FALSE(base->Equals(*deleted));
+  ASSERT_EQ(deleted->num_rows(), base->num_rows() - edit1_->num_rows());
+
+  ASSERT_OK_AND_ASSIGN(
+      auto base_info,
+      WriteAndGetParquetInfo(base, kMinChunkSize, kMaxChunkSize, kEnableDictionary,
+                             ParquetDataPageVersion::V1, kRowGroupLength));
+  ASSERT_OK_AND_ASSIGN(
+      auto deleted_info,
+      WriteAndGetParquetInfo(deleted, kMinChunkSize, kMaxChunkSize, kEnableDictionary,
+                             ParquetDataPageVersion::V1, kRowGroupLength));
+
+  ASSERT_EQ(base_info.size(), 7);
+  ASSERT_EQ(deleted_info.size(), 7);
+
+  ASSERT_EQ(base_info.at(0).page_lengths, deleted_info.at(0).page_lengths);
+  ASSERT_EQ(base_info.at(1).page_lengths, deleted_info.at(1).page_lengths);
+  for (size_t i = 2; i < deleted_info.size() - 1; i++) {
+    AssertPageLengthDifferences(base_info.at(i), deleted_info.at(i),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/1,
+                                /*exact_number_of_smaller_diffs=*/1, edit1_->num_rows());

Review Comment:
   Haha, no :) 
   I will try to give some description to those cases.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027842061


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk

Review Comment:
   Removed it, now it calculated exact chunk sizes accounting for the null entries as well. 



##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk

Review Comment:
   Removed it, now it calculates exact chunk sizes accounting for the null entries as well. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2005306344


##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1332,13 +1337,38 @@ class TypedColumnWriterImpl : public ColumnWriterImpl, public TypedColumnWriter<
       bits_buffer_->ZeroPadding();
     }
 
-    if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
-      return WriteArrowDictionary(def_levels, rep_levels, num_levels, leaf_array, ctx,
-                                  maybe_parent_nulls);
+    if (this->properties_->cdc_enabled()) {
+      ARROW_ASSIGN_OR_RAISE(auto boundaries,
+                            content_defined_chunker_.GetBoundaries(
+                                def_levels, rep_levels, num_levels, leaf_array));
+      for (auto chunk : boundaries) {
+        auto chunk_array = leaf_array.Slice(chunk.value_offset);

Review Comment:
   Theoretically the same value index advancing logic is used in the writer implementations. 
   I can track the length of each value chunk just in case though. Can you think of an example which I could test against?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2847348450

   > (Sorry... I don't have enough bandwidth to review this... I trust wgtmac's review.)
   
   No worries, thanks for your previous reviews!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2005300075


##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1332,13 +1337,38 @@ class TypedColumnWriterImpl : public ColumnWriterImpl, public TypedColumnWriter<
       bits_buffer_->ZeroPadding();
     }
 
-    if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
-      return WriteArrowDictionary(def_levels, rep_levels, num_levels, leaf_array, ctx,
-                                  maybe_parent_nulls);
+    if (properties_->cdc_enabled()) {
+      ARROW_ASSIGN_OR_RAISE(auto boundaries,
+                            content_defined_chunker_.GetBoundaries(

Review Comment:
   > There are some cases where the parquet writer will further split the input Arrow array into smaller pieces which may affect the precision of the CDC logic here:
   > 
   > * Split the input for max_row_group_size: https://github.com/apache/arrow/blob/main/cpp/src/parquet/arrow/writer.cc#L458-L470
   I am aware of this problem, this indeed reduces the efficiency. In case of multiple row groups the row groups following the modifications will have changed pages at the beginning and end of them. 
   
   A proper solution for this problem is to pick a column from the table and calculate row group boundaries based on that single column. I have done some experiments, and the deduplication efficiency drop caused by the fixed sized row group chunking is not significant, though it also depends on the number of pages in a row group.
   I am planning to further experiment with this after we merge this PR.
   
   > * If data page v2 or page index is enabled, page boundary must be a record boundary (i.e. rep_level = 0), this prohibits page cut at certain values: https://github.com/apache/arrow/blob/main/cpp/src/parquet/column_writer.cc#L1142-L1188
   
   The CDC chunker always cuts at record boundaries.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1968434363


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {

Review Comment:
   Here with dictionary encoding disabled:
   
   ![newplot-9](https://github.com/user-attachments/assets/df7d7ed8-62cd-4949-9c08-560bdeb08ccc)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1992134292


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             int64_t min_size, int64_t max_size,
+                                             int8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(min_size),
+      max_size_(max_size),
+      hash_mask_(GetMask(min_size, max_size, norm_factor)) {
+  if (min_size_ < 0) {
+    throw ParquetException("min_size must be non-negative");
+  }
+  if (max_size_ < 0) {
+    throw ParquetException("max_size must be non-negative");
+  }
+  if (min_size_ > max_size_) {
+    throw ParquetException("min_size must be less than or equal to max_size");
+  }
+}
+
+void ContentDefinedChunker::Roll(const bool value) {
+  if (chunk_size_++ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+  has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+}
+
+template <int ByteWidth>
+void ContentDefinedChunker::Roll(const uint8_t* value) {
+  chunk_size_ += ByteWidth;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (size_t i = 0; i < ByteWidth; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T* value) {
+  return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+}
+
+void ContentDefinedChunker::Roll(const uint8_t* value, int64_t length) {
+  chunk_size_ += length;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (auto i = 0; i < length; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, note that we don't reset the
+    // rolling hash state here, so the next NeedNewChunk() call will continue from the
+    // current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename RollFunc>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const RollFunc& RollValue) {
+  std::vector<Chunk> chunks;
+  int64_t offset;
+  int64_t prev_offset = 0;
+  int64_t prev_value_offset = 0;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    for (offset = 0; offset < num_levels; ++offset) {
+      RollValue(offset);
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int16_t def_level;
+    for (int64_t offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+
+      Roll(&def_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(offset);
+      }
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else {
+    // nested data with nulls
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t value_offset = 0;
+
+    for (offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+      rep_level = rep_levels[offset];
+
+      Roll(&def_level);
+      Roll(&rep_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(value_offset);
+      }
+
+      if ((rep_level == 0) && NeedNewChunk()) {
+        // if we are at a record boundary and need a new chunk, we create a new chunk
+        auto levels_to_write = offset - prev_offset;
+        if (levels_to_write > 0) {
+          chunks.emplace_back(prev_offset, prev_value_offset, levels_to_write);
+          prev_offset = offset;
+          prev_value_offset = value_offset;
+        }
+      }
+      if (def_level >= level_info_.repeated_ancestor_def_level) {
+        // we only increment the value offset if we have a leaf value
+        ++value_offset;
+      }
+    }
+  }
+
+  // add the last chunk if we have any levels left
+  if (prev_offset < num_levels) {
+    chunks.emplace_back(prev_offset, prev_value_offset, num_levels - prev_offset);
+  }
+  return chunks;
+}
+
+#define FIXED_WIDTH_CASE(ByteWidth)                                       \
+  {                                                                       \
+    const auto raw_values = values.data()->GetValues<uint8_t>(1);         \

Review Comment:
   Isn't `GetValues()` with a single argument supposed to incorporate the offset?
   
   ```cpp
     template <typename T>
     inline const T* GetValues(int i) const {
       return GetValues<T>(i, offset);
     }
   
     // Like GetValues, but returns NULLPTR instead of aborting if the underlying
     // buffer is not a CPU buffer.
     template <typename T>
     inline const T* GetValuesSafe(int i, int64_t absolute_offset) const {
       if (buffers[i] && buffers[i]->is_cpu()) {
         return reinterpret_cast<const T*>(buffers[i]->data()) + absolute_offset;
       } else {
         return NULLPTR;
       }
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1985561762


##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1332,13 +1337,38 @@ class TypedColumnWriterImpl : public ColumnWriterImpl, public TypedColumnWriter<
       bits_buffer_->ZeroPadding();
     }
 
-    if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
-      return WriteArrowDictionary(def_levels, rep_levels, num_levels, leaf_array, ctx,
-                                  maybe_parent_nulls);
+    if (this->properties_->cdc_enabled()) {
+      ARROW_ASSIGN_OR_RAISE(auto boundaries,
+                            content_defined_chunker_.GetBoundaries(
+                                def_levels, rep_levels, num_levels, leaf_array));
+      for (auto chunk : boundaries) {
+        auto chunk_array = leaf_array.Slice(chunk.value_offset);
+        auto chunk_def_levels = AddIfNotNull(def_levels, chunk.level_offset);
+        auto chunk_rep_levels = AddIfNotNull(rep_levels, chunk.level_offset);
+        if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
+          ARROW_CHECK_OK(WriteArrowDictionary(chunk_def_levels, chunk_rep_levels,
+                                              chunk.levels_to_write, *chunk_array, ctx,
+                                              maybe_parent_nulls));
+        } else {
+          ARROW_CHECK_OK(WriteArrowDense(chunk_def_levels, chunk_rep_levels,
+                                         chunk.levels_to_write, *chunk_array, ctx,
+                                         maybe_parent_nulls));
+        }
+        if (num_buffered_values_ > 0) {
+          AddDataPage();
+        }
+      }
+      return Status::OK();
     } else {
-      return WriteArrowDense(def_levels, rep_levels, num_levels, leaf_array, ctx,
-                             maybe_parent_nulls);
+      if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
+        return WriteArrowDictionary(def_levels, rep_levels, num_levels, leaf_array, ctx,
+                                    maybe_parent_nulls);
+      } else {
+        return WriteArrowDense(def_levels, rep_levels, num_levels, leaf_array, ctx,
+                               maybe_parent_nulls);
+      }

Review Comment:
   No, this is the original non-CDC code path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1988119627


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,250 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_hashtable.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size

Review Comment:
   By increasing the number of gearhash-tables we can smooth out the distribution, but code size would increase too as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1976493385


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+bool ContentDefinedChunker::Check() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last Check() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling
+    // hash state here, so the next Check() call will continue from the current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename T>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const T& leaf_array) {
+  std::vector<Chunk> result;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(def_levels[offset]);
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else {
+    // nested data with nulls
+    bool has_leaf_value;
+    bool is_record_boundary;
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t level_offset = 0;
+    int64_t value_offset = 0;
+    int64_t record_level_offset = 0;
+    int64_t record_value_offset = 0;
+
+    while (level_offset < num_levels) {
+      def_level = def_levels[level_offset];
+      rep_level = rep_levels[level_offset];
+
+      has_leaf_value = def_level >= level_info_.repeated_ancestor_def_level;
+      is_record_boundary = rep_level == 0;
+
+      Roll(def_level);
+      Roll(rep_level);
+      if (has_leaf_value) {
+        Roll(leaf_array.GetView(value_offset));
+      }
+
+      if (is_record_boundary && Check()) {
+        auto levels_to_write = level_offset - record_level_offset;
+        if (levels_to_write > 0) {
+          result.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+          record_level_offset = level_offset;
+          record_value_offset = value_offset;
+        }
+      }
+
+      ++level_offset;
+      if (has_leaf_value) {
+        ++value_offset;
+      }
+    }
+
+    auto levels_to_write = num_levels - record_level_offset;
+    if (levels_to_write > 0) {
+      result.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+    }
+  }
+
+  return result;
+}
+
+#define PRIMITIVE_CASE(TYPE_ID, ArrowType)               \
+  case ::arrow::Type::TYPE_ID:                           \
+    return Calculate(def_levels, rep_levels, num_levels, \
+                     static_cast<const ::arrow::ArrowType##Array&>(values));
+
+const ::arrow::Result<std::vector<Chunk>> ContentDefinedChunker::GetBoundaries(
+    const int16_t* def_levels, const int16_t* rep_levels, int64_t num_levels,
+    const ::arrow::Array& values) {
+  auto type_id = values.type()->id();
+  switch (type_id) {

Review Comment:
   I tried, but makes the implementation way more complicated. 



##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+bool ContentDefinedChunker::Check() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last Check() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling
+    // hash state here, so the next Check() call will continue from the current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename T>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const T& leaf_array) {
+  std::vector<Chunk> result;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(def_levels[offset]);
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else {
+    // nested data with nulls
+    bool has_leaf_value;
+    bool is_record_boundary;
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t level_offset = 0;
+    int64_t value_offset = 0;
+    int64_t record_level_offset = 0;
+    int64_t record_value_offset = 0;
+
+    while (level_offset < num_levels) {
+      def_level = def_levels[level_offset];
+      rep_level = rep_levels[level_offset];
+
+      has_leaf_value = def_level >= level_info_.repeated_ancestor_def_level;
+      is_record_boundary = rep_level == 0;
+
+      Roll(def_level);
+      Roll(rep_level);
+      if (has_leaf_value) {
+        Roll(leaf_array.GetView(value_offset));
+      }
+
+      if (is_record_boundary && Check()) {
+        auto levels_to_write = level_offset - record_level_offset;
+        if (levels_to_write > 0) {
+          result.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+          record_level_offset = level_offset;
+          record_value_offset = value_offset;
+        }
+      }
+
+      ++level_offset;
+      if (has_leaf_value) {
+        ++value_offset;
+      }
+    }
+
+    auto levels_to_write = num_levels - record_level_offset;
+    if (levels_to_write > 0) {
+      result.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+    }
+  }
+
+  return result;
+}
+
+#define PRIMITIVE_CASE(TYPE_ID, ArrowType)               \
+  case ::arrow::Type::TYPE_ID:                           \
+    return Calculate(def_levels, rep_levels, num_levels, \
+                     static_cast<const ::arrow::ArrowType##Array&>(values));
+
+const ::arrow::Result<std::vector<Chunk>> ContentDefinedChunker::GetBoundaries(
+    const int16_t* def_levels, const int16_t* rep_levels, int64_t num_levels,
+    const ::arrow::Array& values) {
+  auto type_id = values.type()->id();
+  switch (type_id) {

Review Comment:
   > I tried, but it makes the implementation way more complicated.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1988087509


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, uint64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal128Builder, ::arrow::Decimal128(val % max_value))
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal256Builder, ::arrow::Decimal256(val % max_value))
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      GENERATE_CASE_BODY(::arrow::FixedSizeBinaryBuilder,
+                         std::string("bin_") + std::to_string(val).substr(0, size - 4))
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, uint64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(const std::shared_ptr<Table>& table,
+                                                   uint64_t min_chunk_size,
+                                                   uint64_t max_chunk_size,
+                                                   bool enable_dictionary = false,
+
+                                                   int64_t row_group_size = 1024 * 1024) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_cdc()
+      ->cdc_size_range(min_chunk_size, max_chunk_size)
+      ->cdc_norm_factor(0);

Review Comment:
   This is my preference as well, updating.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1978943527


##########
cpp/src/parquet/properties.h:
##########
@@ -275,10 +282,33 @@ class PARQUET_EXPORT WriterProperties {
           page_checksum_enabled_(properties.page_checksum_enabled()),
           size_statistics_level_(properties.size_statistics_level()),
           sorting_columns_(properties.sorting_columns()),
-          default_column_properties_(properties.default_column_properties()) {}
+          default_column_properties_(properties.default_column_properties()),
+          cdc_enabled_(properties.cdc_enabled()),
+          cdc_size_range_(properties.cdc_size_range()),
+          cdc_norm_factor_(properties.cdc_norm_factor()) {}
 
     virtual ~Builder() {}
 
+    Builder* enable_cdc() {

Review Comment:
   Agree, it is confusing with change data capture, going to spell out.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kou (via GitHub)" <gi...@apache.org>]

kou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1978173101


##########
cpp/src/parquet/column_chunker_hashtable.py:
##########
@@ -0,0 +1,90 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import hashlib
+import pathlib
+import sys
+from io import StringIO
+
+
+template = """\
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+#include <cstdint>
+
+namespace parquet {{
+namespace internal {{

Review Comment:
   Could you use nested namespaces? See also: https://github.com/apache/arrow/issues/36620
   
   ```suggestion
   namespace parquet::internal {{
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2085268278


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1687 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include <gtest/gtest.h>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::internal::checked_cast;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = checked_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = checked_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size =
+          checked_cast<const ::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = checked_cast<const ::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array, GenerateArray(struct_type->field(i),
+                                                              length, seed + i * 10));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      // Repeat the same pattern in the list array:
+      // null, empty list, list of 1 element, list of 3 elements
+      if (length % 4 != 0) {
+        return Status::Invalid(
+            "Length must be divisible by 4 when generating list arrays, but got: ",
+            length);
+      }
+      auto values_array_length = length * 4;
+      auto list_type = checked_cast<const ::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array,
+                            GenerateArray(value_field, values_array_length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      RETURN_NOT_OK(offset_builder.Reserve(length + 1));
+      RETURN_NOT_OK(bitmap_builder.Reserve(length));
+
+      int32_t num_nulls = 0;
+      RETURN_NOT_OK(offset_builder.Append(0));
+      for (auto offset = 0; offset < length; offset += 4) {
+        if (nullable) {
+          // add a null
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          RETURN_NOT_OK(offset_builder.Append(offset));
+          num_nulls += 1;
+        } else {
+          // add an empty list
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          RETURN_NOT_OK(offset_builder.Append(offset));
+        }
+        // add an empty list
+        RETURN_NOT_OK(bitmap_builder.Append(true));
+        RETURN_NOT_OK(offset_builder.Append(offset));
+        // add a list of 1 element
+        RETURN_NOT_OK(bitmap_builder.Append(true));
+        RETURN_NOT_OK(offset_builder.Append(offset + 1));
+        // add a list of 3 elements
+        RETURN_NOT_OK(bitmap_builder.Append(true));
+        RETURN_NOT_OK(offset_builder.Append(offset + 4));
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = checked_cast<const ::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+TEST(TestGenerateArray, Integer) {
+  auto field = ::arrow::field("a", ::arrow::int32());
+  ASSERT_OK_AND_ASSIGN(auto array, GenerateArray(field, /*length=*/10, /*seed=*/0));
+  ASSERT_OK(array->ValidateFull());
+  ASSERT_EQ(array->length(), 10);
+  ASSERT_TRUE(array->type()->Equals(::arrow::int32()));
+  ASSERT_EQ(array->null_count(), 1);
+}
+
+TEST(TestGenerateArray, ListOfInteger) {
+  auto field = ::arrow::field("a", ::arrow::list(::arrow::int32()));
+  auto length = 12;
+  ASSERT_OK_AND_ASSIGN(auto array, GenerateArray(field, length, /*seed=*/0));
+  ASSERT_OK(array->ValidateFull());
+  ASSERT_EQ(array->length(), length);
+
+  for (size_t i = 0; i < 12; i += 4) {
+    // Assert the first element is null
+    ASSERT_TRUE(array->IsNull(i));
+
+    // Assert the second element is an empty list
+    ASSERT_TRUE(array->IsValid(i + 1));
+    auto list_array = std::static_pointer_cast<::arrow::ListArray>(array);
+    ASSERT_EQ(list_array->value_length(i + 1), 0);
+
+    // Assert the third element has length 1
+    ASSERT_TRUE(array->IsValid(i + 2));
+    ASSERT_EQ(list_array->value_length(i + 2), 1);
+
+    // Assert the fourth element has length 3
+    ASSERT_TRUE(array->IsValid(i + 3));
+    ASSERT_EQ(list_array->value_length(i + 3), 3);
+  }
+
+  ASSERT_NOT_OK(GenerateArray(field, 3, /*seed=*/0));
+  ASSERT_OK(GenerateArray(field, 8, /*seed=*/0));
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, ++seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  auto props = default_arrow_reader_properties();
+
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(props)
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    int64_t row_group_length = 1024 * 1024, bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      {min_chunk_size, max_chunk_size, /*norm_level=*/0});
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_length,
+                           write_props, arrow_props));
+  ARROW_ASSIGN_OR_RAISE(auto buffer, sink->Finish());
+
+  // check whether the schema has extension types, if not we can easily ensure that
+  // the parquet seralization is roundtripable with CDC enabled
+  bool validate_roundtrip = true;
+  for (const auto& field : table->schema()->fields()) {
+    if (field->type()->id() == ::arrow::Type::EXTENSION) {
+      validate_roundtrip = false;

Review Comment:
   The test case used `UUID` which isn't supported by the parquet arrow reader:
   
   ```cpp
     /// Enable Parquet-supported Arrow extension types.
     ///
     /// When enabled, Parquet logical types will be mapped to their corresponding Arrow
     /// extension types at read time, if such exist. Currently only arrow::extension::json()
     /// extension type is supported. Columns whose LogicalType is JSON will be interpreted
     /// as arrow::extension::json(), with storage type inferred from the serialized Arrow
     /// schema if present, or `utf8` by default.
     void set_arrow_extensions_enabled(bool extensions_enabled) {
       arrow_extensions_enabled_ = extensions_enabled;
     }
   ```
   
   I changed the test case to use `arrow::extension::json()` instead, it tests the same functionality.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kou (via GitHub)" <gi...@apache.org>]

kou commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2847345463

   (Sorry... I don't have enough bandwidth to review this... I trust wgtmac's review.)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2745241706

   It would be nice to place some information about the used CDC chunking approach in the arbitrary key-value metadata. Do we have any sort of preference there? Should we automatically add it or let the user manually define it?
   An example I could think of: `"CDC" => {"min_chunk_size": ..., "max_chunk_size": ..., "norm_factor": ...}`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2016989421


##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1382,11 +1383,15 @@ class TypedColumnWriterImpl : public ColumnWriterImpl,
                                          chunk.levels_to_write, *chunk_array, ctx,
                                          maybe_parent_nulls));
         }
-        if (num_buffered_values_ > 0) {
+        bool is_last_chunk = i == chunks.size() - 1;
+        if (num_buffered_values_ > 0 && !is_last_chunk) {
           // Explicitly add a new data page according to the content-defined chunk
           // boundaries. This way the same chunks will have the same byte-sequence
           // in the resulting file, which can be identified by content addressible
           // storage.
+          // Note that the last chunk doesn't trigger a new data page in order to
+          // allow subsequent WriteArrow() calls to continue writing to the same
+          // data page, the chunker's state is not being reset after the last chunk.

Review Comment:
   The chunker post-checks max size after adding the value to the chunk, if it exceeds the max chunk size then we don't add more values to that chunk but rather split. This is similar to how `max_data_page_size` is being checked.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["mapleFU (via GitHub)" <gi...@apache.org>]

mapleFU commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2013586539


##########
cpp/src/parquet/properties.h:
##########
@@ -260,7 +268,9 @@ class PARQUET_EXPORT WriterProperties {
           created_by_(DEFAULT_CREATED_BY),
           store_decimal_as_integer_(false),
           page_checksum_enabled_(false),
-          size_statistics_level_(DEFAULT_SIZE_STATISTICS_LEVEL) {}
+          size_statistics_level_(DEFAULT_SIZE_STATISTICS_LEVEL),
+          content_defined_chunking_enabled_(false),

Review Comment:
   I'm think of would a `optional<CDCOptions>` better here?



##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+#include "parquet/properties.h"
+
+namespace parquet::internal {
+
+class TestPina;
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously

Review Comment:
   ```suggestion
   /// modifications. Each chunk defines a new parquet data page which is contiguously
   ```



##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>

Review Comment:
   switch line?



##########
cpp/src/parquet/properties.h:
##########
@@ -245,6 +245,14 @@ class PARQUET_EXPORT ColumnProperties {
   bool page_index_enabled_;
 };
 
+struct CDCOptions {

Review Comment:
   Do we have a "validate" or check for CDCOptions? what if negative size passed in or `min_chunk_size > max_chunk_size`



##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+#include "parquet/properties.h"
+
+namespace parquet::internal {
+
+class TestPina;
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage system (CAS)
+/// which splits the bytes stream into content defined blobs. The CAS system will
+/// calculate a unique identifier for each blob, then store the blob in a key-value store.
+/// If the same blob is encountered again, the system can refer to the hash instead of
+/// physically storing the blob again. In the example above, the CAS system would store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks.
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data
+///   Deduplication
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+/// - Git is for Data (chunk size normalization used here is described in section 6.2.1):
+///   https://www.cidrdb.org/cidr2023/papers/p43-low.pdf
+class PARQUET_EXPORT ContentDefinedChunker {
+ public:
+  /// Create a new ContentDefinedChunker instance
+  ///
+  /// @param level_info Information about definition and repetition levels
+  /// @param min_chunk_size Minimum chunk size in bytes
+  ///   The rolling hash will not be updated until this size is reached for each chunk.
+  ///   Note that all data sent through the hash function is counted towards the chunk
+  ///   size, including definition and repetition levels if present.
+  /// @param max_chunk_size Maximum chunk size in bytes
+  ///   The chunker creates a new chunk whenever the chunk size exceeds this value. The
+  ///   chunk size distribution approximates a normal distribution between min_chunk_size
+  ///   and max_chunk_size. Note that the parquet writer has a related `data_pagesize`
+  //    property that controls the maximum size of a parquet data page after encoding.
+  ///   While setting `data_pagesize` to a smaller value than `max_chunk_size` doesn't
+  ///   affect the chunking effectiveness, it results in more small parquet data pages.
+  /// @param norm_factor Normalization factor to center the chunk size around the average
+  ///   size more aggressively, default 0.
+  ///   Increasing the normalization factor increases the probability of finding a chunk
+  ///   boundary, improving the deduplication ratio, but also increases the number of
+  ///   small chunks resulting in many small parquet data pages. The default value
+  ///   provides a good balance between deduplication ratio and fragmentation.
+  ///   Use norm_factor=1 or norm_factor=2 to reach a higher deduplication ratio at the
+  ///   expense of fragmentation.
+  ContentDefinedChunker(const LevelInfo& level_info, int64_t min_chunk_size,
+                        int64_t max_chunk_size, int8_t norm_factor = 0);
+  ~ContentDefinedChunker();
+
+  /// Get the chunk boundaries for the given column data

Review Comment:
   Can we guarantees the returned `vector<Chunks>` successive? And when it comes to the small batch or final chunks, what would it returns? An empty `vector<Chunk>`?



##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+#include "parquet/properties.h"
+
+namespace parquet::internal {
+
+class TestPina;
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage system (CAS)

Review Comment:
   ```suggestion
   /// Then the parquet file is being uploaded to a content addressable storage (CAS) system 
   ```
   
   ?



##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1332,13 +1337,38 @@ class TypedColumnWriterImpl : public ColumnWriterImpl, public TypedColumnWriter<
       bits_buffer_->ZeroPadding();
     }
 
-    if (leaf_array.type()->id() == ::arrow::Type::DICTIONARY) {
-      return WriteArrowDictionary(def_levels, rep_levels, num_levels, leaf_array, ctx,
-                                  maybe_parent_nulls);
+    if (this->properties_->cdc_enabled()) {
+      ARROW_ASSIGN_OR_RAISE(auto boundaries,
+                            content_defined_chunker_.GetBoundaries(
+                                def_levels, rep_levels, num_levels, leaf_array));
+      for (auto chunk : boundaries) {
+        auto chunk_array = leaf_array.Slice(chunk.value_offset);

Review Comment:
   should we dcheck the internal chunks grows incremental?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2019759018


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int8_t norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes
+  int64_t target_size = avg_chunk_size - min_chunk_size;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // 3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  if (norm_factor < -3 || norm_factor > 3) {
+    ARROW_LOG(WARNING) << "norm_factor=" << std::to_string(norm_factor)
+                       << " is outside the recommended range [-3, 3]";
+  }
+
+  int mask_adjustement = 3 + norm_factor;
+  int effective_bits = mask_bits - mask_adjustement;
+  if (effective_bits == 0) {
+    return 0;
+  } else if (effective_bits > 64) {
+    throw ParquetException("The number of bits in the mask cannot exceed 64, got " +
+                           std::to_string(effective_bits));
+  } else if (effective_bits < 0) {
+    throw ParquetException(
+        "The difference between min_chunk_size=" + std::to_string(min_chunk_size) +
+        " and max_chunk_size=" + std::to_string(max_chunk_size) +
+        " is too small for the given norm_factor=" + std::to_string(norm_factor) +
+        ", either increase the size range or decrease the norm_factor.");
+  } else {
+    // create the mask by setting the top mask_bits bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int8_t norm_factor)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_factor)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(const bool value) {
+    if (chunk_size_++ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+    has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+  }
+
+  template <int ByteWidth>
+  void Roll(const uint8_t* value) {
+    // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+    // true if the hash matches the mask.
+
+    chunk_size_ += ByteWidth;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (size_t i = 0; i < ByteWidth; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  template <typename T>
+  void Roll(const T* value) {
+    return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+  }
+
+  void Roll(const uint8_t* value, int64_t length) {
+    // Update the rolling hash with a binary-like value, set has_matched_ to true if the
+    // hash matches the mask.
+
+    chunk_size_ += length;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (auto i = 0; i < length; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  bool NeedNewChunk() {
+    // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+    // set to true if we encountered a match since the last NeedNewChunk() call
+    if (ARROW_PREDICT_FALSE(has_matched_)) {
+      has_matched_ = false;
+      // in order to have a normal distribution of chunk sizes, we only create a new chunk
+      // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+      // different gearhash table (gearhash's chunk size has geometric distribution, and
+      // we use central limit theorem to approximate normal distribution, see
+      // section 6.2.1 in paper https://www.cidrdb.org/cidr2023/papers/p43-low.pdf)
+      if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+        nth_run_ = 0;
+        chunk_size_ = 0;
+        return true;
+      }
+    }
+    if (ARROW_PREDICT_FALSE(chunk_size_ >= max_chunk_size_)) {
+      // we have a hard limit on the maximum chunk size, note that we don't reset the
+      // rolling hash state here, so the next NeedNewChunk() call will continue from the
+      // current state
+      chunk_size_ = 0;
+      return true;
+    }
+    return false;
+  }
+
+  void ValidateChunks(const std::vector<Chunk>& chunks, int64_t num_levels) {

Review Comment:
   Although `ValidateChunks` uses debug checks, it still has a loop which would be compiled in release builds. I am sparing that loop with the `NDEBUG` check to not call `ValidateChunks()` at all. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1998841562


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,250 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_hashtable.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][bytes[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling
+    // hash state here, so the next NeedNewChunk() call will continue from the current
+    // state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename T>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const T& leaf_array) {
+  std::vector<Chunk> chunks;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      chunks.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(def_levels[offset]);
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      chunks.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else {
+    // nested data with nulls
+    bool has_leaf_value;
+    bool is_record_boundary;
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t value_offset = 0;
+    int64_t record_level_offset = 0;
+    int64_t record_value_offset = 0;
+
+    for (int64_t level_offset = 0; level_offset < num_levels; ++level_offset) {
+      def_level = def_levels[level_offset];
+      rep_level = rep_levels[level_offset];
+
+      has_leaf_value = def_level >= level_info_.repeated_ancestor_def_level;
+      is_record_boundary = rep_level == 0;
+
+      Roll(def_level);
+      Roll(rep_level);
+      if (has_leaf_value) {
+        Roll(leaf_array.GetView(value_offset));
+      }
+
+      if (is_record_boundary && NeedNewChunk()) {
+        auto levels_to_write = level_offset - record_level_offset;
+        if (levels_to_write > 0) {
+          chunks.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+          record_level_offset = level_offset;
+          record_value_offset = value_offset;
+        }
+      }
+
+      if (has_leaf_value) {
+        ++value_offset;
+      }
+    }
+
+    auto levels_to_write = num_levels - record_level_offset;
+    if (levels_to_write > 0) {
+      chunks.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+    }
+  }
+
+  return chunks;
+}
+
+#define PRIMITIVE_CASE(TYPE_ID, ArrowType)               \
+  case ::arrow::Type::TYPE_ID:                           \
+    return Calculate(def_levels, rep_levels, num_levels, \
+                     static_cast<const ::arrow::ArrowType##Array&>(values));
+
+const std::vector<Chunk> ContentDefinedChunker::GetBoundaries(
+    const int16_t* def_levels, const int16_t* rep_levels, int64_t num_levels,
+    const ::arrow::Array& values) {
+  auto type_id = values.type()->id();
+  switch (type_id) {
+    PRIMITIVE_CASE(BOOL, Boolean)
+    PRIMITIVE_CASE(INT8, Int8)
+    PRIMITIVE_CASE(INT16, Int16)

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2026597313


##########
cpp/src/parquet/properties.h:
##########
@@ -245,6 +245,14 @@ class PARQUET_EXPORT ColumnProperties {
   bool page_index_enabled_;
 };
 
+struct CDCOptions {
+  int64_t min_chunk_size;
+  int64_t max_chunk_size;
+  int8_t norm_factor;
+};
+
+static constexpr CDCOptions kDefaultCdcOptions = CDCOptions{256 * 1024, 1024 * 1024, 0};

Review Comment:
   No worries, renaming.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1995707740


##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -390,6 +391,7 @@ add_parquet_test(reader-test
 
 add_parquet_test(writer-test
                  SOURCES
+                 column_chunker_test.cc

Review Comment:
   Pulled out to a separate testing binary.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025073961


##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <vector>
+
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which is contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage (CAS) system
+/// which splits the bytes stream into content defined blobs. The CAS system will
+/// calculate a unique identifier for each blob, then store the blob in a key-value store.
+/// If the same blob is encountered again, the system can refer to the hash instead of
+/// physically storing the blob again. In the example above, the CAS system would store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks.
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data
+///   Deduplication
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+/// - Git is for Data (chunk size normalization used here is described in section 6.2.1):
+///   https://www.cidrdb.org/cidr2023/papers/p43-low.pdf
+class PARQUET_EXPORT ContentDefinedChunker {
+ public:
+  /// Create a new ContentDefinedChunker instance
+  ///
+  /// @param level_info Information about definition and repetition levels
+  /// @param min_chunk_size Minimum chunk size in bytes
+  ///   The rolling hash will not be updated until this size is reached for each chunk.
+  ///   Note that all data sent through the hash function is counted towards the chunk
+  ///   size, including definition and repetition levels if present.
+  /// @param max_chunk_size Maximum chunk size in bytes
+  ///   The chunker creates a new chunk whenever the chunk size exceeds this value. The
+  ///   chunk size distribution approximates a normal distribution between min_chunk_size
+  ///   and max_chunk_size. Note that the parquet writer has a related `data_pagesize`
+  //    property that controls the maximum size of a parquet data page after encoding.
+  ///   While setting `data_pagesize` to a smaller value than `max_chunk_size` doesn't
+  ///   affect the chunking effectiveness, it results in more small parquet data pages.
+  /// @param norm_factor Normalization factor to center the chunk size around the average
+  ///   size more aggressively, default 0.
+  ///   Increasing the normalization factor increases the probability of finding a chunk
+  ///   boundary, improving the deduplication ratio, but also increases the number of
+  ///   small chunks resulting in many small parquet data pages. The default value
+  ///   provides a good balance between deduplication ratio and fragmentation.
+  ///   Use norm_factor=1 or norm_factor=2 to reach a higher deduplication ratio at the
+  ///   expense of fragmentation.
+  ContentDefinedChunker(const LevelInfo& level_info, int64_t min_chunk_size,
+                        int64_t max_chunk_size, int8_t norm_factor = 0);

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025067327


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int8_t norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes
+  int64_t target_size = avg_chunk_size - min_chunk_size;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // 3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  if (norm_factor < -3 || norm_factor > 3) {
+    ARROW_LOG(WARNING) << "norm_factor=" << norm_factor
+                       << " is outside the recommended range [-3, 3]";
+  }
+
+  int mask_adjustement = 3 + norm_factor;

Review Comment:
   Moved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025170579


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025157413


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025192192


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();
+    auto min_length = ElementCount(min_chunk_size, byte_width, nullable);
+    auto max_length = ElementCount(max_chunk_size, byte_width, nullable);
+    AssertAllBetween(base_info.page_lengths, min_length, max_length);
+    AssertAllBetween(modified_info.page_lengths, min_length, max_length);
+  } else if (::arrow::is_base_binary_like(dtype->id()) && !enable_dictionary) {
+    AssertAllBetween(base_info.page_sizes, min_chunk_size, max_chunk_size);
+    AssertAllBetween(modified_info.page_sizes, min_chunk_size, max_chunk_size);
+  }
+}
+
+constexpr int64_t kMinChunkSize = 8 * 1024;
+constexpr int64_t kMaxChunkSize = 32 * 1024;
+constexpr int64_t kPartSize = 128 * 1024;
+constexpr int64_t kEditSize = 128;
+
+struct CaseConfig {
+  // Arrow data type to generate the testing data for
+  std::shared_ptr<::arrow::DataType> dtype;
+  // Whether the data type is nullable
+  bool is_nullable;
+  // Approximate number of bytes per record to calculate the number of elements to
+  // generate
+  size_t bytes_per_record;

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025284496


##########
cpp/src/parquet/column_writer.cc:
##########
@@ -892,6 +897,8 @@ class ColumnWriterImpl {
 
   std::vector<std::unique_ptr<DataPage>> data_pages_;
 
+  internal::ContentDefinedChunker content_defined_chunker_;

Review Comment:
   Using optional now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025315563


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int8_t norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes
+  int64_t target_size = avg_chunk_size - min_chunk_size;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // 3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  if (norm_factor < -3 || norm_factor > 3) {
+    ARROW_LOG(WARNING) << "norm_factor=" << norm_factor
+                       << " is outside the recommended range [-3, 3]";
+  }
+
+  int mask_adjustement = 3 + norm_factor;
+  int effective_bits = mask_bits - mask_adjustement;
+  if (effective_bits == 0) {
+    return 0;
+  } else if (effective_bits > 64) {
+    throw ParquetException("The number of bits in the mask cannot exceed 64, got " +
+                           std::to_string(effective_bits));
+  } else if (effective_bits < 0) {
+    throw ParquetException(
+        "The difference between min_chunk_size=" + std::to_string(min_chunk_size) +
+        " and max_chunk_size=" + std::to_string(max_chunk_size) +
+        " is too small for the given norm_factor=" + std::to_string(norm_factor) +
+        ", either increase the size range or decrease the norm_factor.");
+  } else {
+    // create the mask by setting the top mask_bits bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int8_t norm_factor)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_factor)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(const bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+    has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+  }
+
+  template <int ByteWidth>
+  void Roll(const uint8_t* value) {
+    // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+    // true if the hash matches the mask.
+
+    chunk_size_ += ByteWidth;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (size_t i = 0; i < ByteWidth; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  template <typename T>
+  void Roll(const T* value) {
+    return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+  }
+
+  void Roll(const uint8_t* value, int64_t length) {
+    // Update the rolling hash with a binary-like value, set has_matched_ to true if the
+    // hash matches the mask.
+
+    chunk_size_ += length;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (auto i = 0; i < length; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  bool NeedNewChunk() {
+    // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+    // set to true if we encountered a match since the last NeedNewChunk() call
+    if (ARROW_PREDICT_FALSE(has_matched_)) {
+      has_matched_ = false;
+      // in order to have a normal distribution of chunk sizes, we only create a new chunk
+      // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+      // different gearhash table (gearhash's chunk size has geometric distribution, and
+      // we use central limit theorem to approximate normal distribution, see
+      // section 6.2.1 in paper https://www.cidrdb.org/cidr2023/papers/p43-low.pdf)
+      if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {

Review Comment:
   I probably meant `++nth_run_ > 7`, updating and using `kNumGearhashTables`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027832305


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();
+    auto min_length = ElementCount(min_chunk_size, byte_width, nullable);
+    auto max_length = ElementCount(max_chunk_size, byte_width, nullable);
+    AssertAllBetween(base_info.page_lengths, min_length, max_length);
+    AssertAllBetween(modified_info.page_lengths, min_length, max_length);
+  } else if (::arrow::is_base_binary_like(dtype->id()) && !enable_dictionary) {
+    AssertAllBetween(base_info.page_sizes, min_chunk_size, max_chunk_size);
+    AssertAllBetween(modified_info.page_sizes, min_chunk_size, max_chunk_size);
+  }
+}
+
+constexpr int64_t kMinChunkSize = 8 * 1024;
+constexpr int64_t kMaxChunkSize = 32 * 1024;
+constexpr int64_t kPartSize = 128 * 1024;
+constexpr int64_t kEditSize = 128;
+
+struct CaseConfig {
+  // Arrow data type to generate the testing data for
+  std::shared_ptr<::arrow::DataType> dtype;
+  // Whether the data type is nullable
+  bool is_nullable;
+  // Approximate number of bytes per record to calculate the number of elements to

Review Comment:
   Yes. I removed this field from CaseConfig and moved the record size estimation to the test case `SetUp()`. The fixed width types can be explicitly calculated and using a constant for the variable sized types.
   
   Currently this is only required to reduce the number of generated samples to avoid the tests heavily slowing down, added a comment about this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027841577


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)

Review Comment:
   Reworked it, now it properly calculates the cdc chunk size based on the page lengths and the original logical array. This is only applicable to the fixed width types, for list-like types I would need to reimplement the logic from the CDC chunker which wouldn't make a lot of sense. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027883648


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int8_t norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes
+  int64_t target_size = avg_chunk_size - min_chunk_size;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // 3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  if (norm_factor < -3 || norm_factor > 3) {
+    ARROW_LOG(WARNING) << "norm_factor=" << norm_factor
+                       << " is outside the recommended range [-3, 3]";
+  }
+
+  int mask_adjustement = 3 + norm_factor;
+  int effective_bits = mask_bits - mask_adjustement;
+  if (effective_bits == 0) {
+    return 0;
+  } else if (effective_bits > 64) {
+    throw ParquetException("The number of bits in the mask cannot exceed 64, got " +
+                           std::to_string(effective_bits));
+  } else if (effective_bits < 0) {
+    throw ParquetException(
+        "The difference between min_chunk_size=" + std::to_string(min_chunk_size) +
+        " and max_chunk_size=" + std::to_string(max_chunk_size) +
+        " is too small for the given norm_factor=" + std::to_string(norm_factor) +
+        ", either increase the size range or decrease the norm_factor.");
+  } else {
+    // create the mask by setting the top mask_bits bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int8_t norm_factor)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_factor)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(const bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+    has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+  }
+
+  template <int ByteWidth>
+  void Roll(const uint8_t* value) {
+    // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+    // true if the hash matches the mask.
+
+    chunk_size_ += ByteWidth;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (size_t i = 0; i < ByteWidth; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  template <typename T>
+  void Roll(const T* value) {
+    return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+  }
+
+  void Roll(const uint8_t* value, int64_t length) {
+    // Update the rolling hash with a binary-like value, set has_matched_ to true if the
+    // hash matches the mask.
+
+    chunk_size_ += length;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (auto i = 0; i < length; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  bool NeedNewChunk() {
+    // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+    // set to true if we encountered a match since the last NeedNewChunk() call
+    if (ARROW_PREDICT_FALSE(has_matched_)) {
+      has_matched_ = false;

Review Comment:
   It is rather a choice. I tried to reset it when we have a match or when we exceed the max chunk size, in practice this doesn't matter much, slightly changes the chunk size distribution. Added a comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027900660


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();
+    auto min_length = ElementCount(min_chunk_size, byte_width, nullable);
+    auto max_length = ElementCount(max_chunk_size, byte_width, nullable);
+    AssertAllBetween(base_info.page_lengths, min_length, max_length);
+    AssertAllBetween(modified_info.page_lengths, min_length, max_length);
+  } else if (::arrow::is_base_binary_like(dtype->id()) && !enable_dictionary) {
+    AssertAllBetween(base_info.page_sizes, min_chunk_size, max_chunk_size);
+    AssertAllBetween(modified_info.page_sizes, min_chunk_size, max_chunk_size);
+  }
+}
+
+constexpr int64_t kMinChunkSize = 8 * 1024;
+constexpr int64_t kMaxChunkSize = 32 * 1024;
+constexpr int64_t kPartSize = 128 * 1024;
+constexpr int64_t kEditSize = 128;
+
+struct CaseConfig {
+  // Arrow data type to generate the testing data for
+  std::shared_ptr<::arrow::DataType> dtype;
+  // Whether the data type is nullable
+  bool is_nullable;
+  // Approximate number of bytes per record to calculate the number of elements to
+  // generate
+  size_t bytes_per_record;
+  // Data page version to use
+  ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1;
+};
+
+// Define PrintTo for MyStruct
+void PrintTo(const CaseConfig& param, std::ostream* os) {
+  *os << "{ " << param.dtype->ToString();
+  if (param.is_nullable) {
+    *os << " nullable";
+  }
+  *os << " }";
+}
+
+class TestCDC : public ::testing::Test {
+ public:
+  uint64_t GetMask(const ContentDefinedChunker& cdc) const { return cdc.GetMask(); }
+};
+
+TEST_F(TestCDC, RollingHashMaskCalculation) {
+  auto le = LevelInfo();
+  auto min_size = 256 * 1024;
+  auto max_size = 1024 * 1024;
+
+  auto cdc0 = ContentDefinedChunker(le, min_size, max_size, 0);
+  ASSERT_EQ(GetMask(cdc0), 0xFFFE000000000000);
+
+  auto cdc1 = ContentDefinedChunker(le, min_size, max_size, 1);
+  ASSERT_EQ(GetMask(cdc1), 0xFFFC000000000000);
+
+  auto cdc2 = ContentDefinedChunker(le, min_size, max_size, 2);
+  ASSERT_EQ(GetMask(cdc2), 0xFFF8000000000000);
+
+  auto cdc3 = ContentDefinedChunker(le, min_size, max_size, 3);
+  ASSERT_EQ(GetMask(cdc3), 0xFFF0000000000000);
+
+  auto cdc4 = ContentDefinedChunker(le, min_size, max_size, -1);
+  ASSERT_EQ(GetMask(cdc4), 0xFFFF000000000000);
+
+  // this is the smallest possible mask always matching, by using 8 hashtables
+  // we are going to have a match every 8 bytes; this is an unrealistic case
+  // but checking for the correctness of the mask calculation
+  auto cdc5 = ContentDefinedChunker(le, 0, 16, 0);
+  ASSERT_EQ(GetMask(cdc5), 0x0000000000000000);
+
+  auto cdc6 = ContentDefinedChunker(le, 0, 32, 1);
+  ASSERT_EQ(GetMask(cdc6), 0x0000000000000000);
+
+  auto cdc7 = ContentDefinedChunker(le, 0, 16, -1);
+  ASSERT_EQ(GetMask(cdc7), 0x8000000000000000);
+
+  // another unrealistic case, checking for the validation
+  auto cdc8 = ContentDefinedChunker(le, 128, 384, -60);
+  ASSERT_EQ(GetMask(cdc8), 0xFFFFFFFFFFFFFFFF);
+}
+
+TEST_F(TestCDC, WriteSingleColumnParquetFile) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  std::vector<int32_t> numbers = {1, 2, 3, 4, 5};
+  std::vector<uint8_t> valid_bits = {1, 0, 1, 0, 1};
+  EXPECT_THROW(
+      int_column_writer.WriteBatch(numbers.size(), nullptr, nullptr, numbers.data()),
+      ParquetException);
+  EXPECT_THROW(int_column_writer.WriteBatchSpaced(numbers.size(), nullptr, nullptr,
+                                                  valid_bits.data(), 0, numbers.data()),
+               ParquetException);
+}
+
+TEST_F(TestCDC, LastChunkDoesntTriggerAddDataPage) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()
+                   ->content_defined_chunking_options(kMinChunkSize, kMaxChunkSize, 0)
+                   ->disable_dictionary()
+                   ->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  ASSERT_OK_AND_ASSIGN(auto array, ::arrow::gen::Step()->Generate(8000));
+  auto arrow_props = default_arrow_writer_properties();
+  auto arrow_ctx = ArrowWriteContext(default_memory_pool(), arrow_props.get());
+
+  // Calling WriteArrow twice, we expect that the first call doesn't add a new data page
+  // at the end allowing subsequent calls to append to the same page
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+
+  int_column_writer.Close();
+  writer->Close();
+  ASSERT_OK_AND_ASSIGN(auto buffer, sink->Finish());
+
+  auto info = GetColumnParquetInfo(buffer);
+  ASSERT_EQ(info.size(), 1);
+
+  // AssertAllBetween allow the last chunk size to be smaller than the min_chunk_size
+  AssertAllBetween(info[0].page_sizes, kMinChunkSize, kMaxChunkSize);
+  AssertAllBetween(info[0].page_lengths, 3000, 5000);
+}
+
+TEST_F(TestCDC, ChunkSizeParameterValidation) {
+  // Test that constructor validates min/max chunk size parameters
+  auto li = LevelInfo();
+
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 256 * 1024, 1024 * 1024));
+
+  // with norm_factor=0 the difference between min and max chunk size must be
+  // at least 16
+  ASSERT_THROW(ContentDefinedChunker(li, 0, -1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1024, 512), ParquetException);
+
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16));
+  ASSERT_THROW(ContentDefinedChunker(li, -16, -16), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 16, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 32, 32), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 32, 48));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1024 * 1024, 2 * 1024 * 1024));
+  ASSERT_NO_THROW(
+      ContentDefinedChunker(li, 1024 * 1024 * 1024L, 2LL * 1024 * 1024 * 1024L));
+
+  // with norm_factor=1 the difference between min and max chunk size must be
+  // at least 64
+  ASSERT_THROW(ContentDefinedChunker(li, 1, -1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 32, 1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1, 33, 1));
+
+  // with norm_factor=2 the difference between min and max chunk size must be
+  // at least 128
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 63, 2), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 64, 2));
+
+  // with norm_factor=-1 the difference between min and max chunk size must be
+  // at least 8
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 7, -1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 8, -1));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16, -1));
+
+  // test the norm_factor extremes
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0, -68), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 1, -67));
+  ASSERT_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 59),
+               ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 58));
+}
+
+class TestCDCSingleRowGroup : public ::testing::TestWithParam<CaseConfig> {
+ protected:
+  // Column random table parts for testing
+  std::shared_ptr<Field> field_;
+  std::shared_ptr<Table> part1_, part2_, part3_, part4_, part5_, part6_, part7_;
+
+  void SetUp() override {
+    const auto& param = GetParam();
+    auto field_ = ::arrow::field("f0", param.dtype, param.is_nullable);
+    auto schema = ::arrow::schema({field_});
+
+    auto part_length = kPartSize / param.bytes_per_record;
+    auto edit_length = kEditSize / param.bytes_per_record;
+    ASSERT_OK_AND_ASSIGN(part1_, GenerateTable(schema, part_length, 0));
+    ASSERT_OK_AND_ASSIGN(part2_, GenerateTable(schema, edit_length, 1));
+    ASSERT_OK_AND_ASSIGN(part3_, GenerateTable(schema, part_length, part_length));
+    ASSERT_OK_AND_ASSIGN(part4_, GenerateTable(schema, edit_length, 2));
+    ASSERT_OK_AND_ASSIGN(part5_, GenerateTable(schema, part_length, 2 * part_length));
+    ASSERT_OK_AND_ASSIGN(part6_, GenerateTable(schema, edit_length, 3));
+    ASSERT_OK_AND_ASSIGN(part7_, GenerateTable(schema, edit_length, 4));
+  }
+};
+
+TEST_P(TestCDCSingleRowGroup, DeleteOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/0,
+                                /*exact_number_of_smaller_diffs=*/1, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, DeleteTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part3_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/0,
+                                /*exact_number_of_smaller_diffs=*/2, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, UpdateOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part4_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*max_number_of_equal_diffs=*/1);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, UpdateTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified,
+                       ConcatAndCombine({part1_, part6_, part3_, part7_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*max_number_of_equal_diffs=*/2);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, InsertOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/1,
+                                /*exact_number_of_smaller_diffs=*/0, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, InsertTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part3_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/2,
+                                /*exact_number_of_smaller_diffs=*/0, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, Append) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part2_, part3_, part4_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto original_page_lengths = base_info.front().page_lengths;
+    auto modified_page_lengths = modified_info.front().page_lengths;
+    ASSERT_GE(original_page_lengths.size(), modified_page_lengths.size());
+    for (size_t i = 0; i < original_page_lengths.size() - 1; i++) {
+      ASSERT_EQ(original_page_lengths[i], modified_page_lengths[i]);
+    }
+    ASSERT_GT(modified_page_lengths.back(), original_page_lengths.back());
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, EmptyTable) {
+  const auto& param = GetParam();
+
+  auto schema = ::arrow::schema({::arrow::field("f0", param.dtype, param.is_nullable)});
+  ASSERT_OK_AND_ASSIGN(auto empty_table, GenerateTable(schema, 0, 0));
+  ASSERT_EQ(empty_table->num_rows(), 0);
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(
+        auto result, WriteAndGetParquetInfo(empty_table, kMinChunkSize, kMaxChunkSize,
+                                            enable_dictionary, param.data_page_version));
+
+    // An empty table should result in no data pages
+    ASSERT_EQ(result.size(), 1);
+    ASSERT_TRUE(result.front().page_lengths.empty());
+    ASSERT_TRUE(result.front().page_sizes.empty());
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, ArrayOffsets) {
+  const auto& param = GetParam();
+  ASSERT_OK_AND_ASSIGN(auto table, ConcatAndCombine({part1_, part2_, part3_}));
+
+  for (auto offset : {0, 512, 1024}) {
+    auto sliced_table = table->Slice(offset);
+
+    // assert that the first column has a non-zero offset
+    auto column = sliced_table->column(0);
+    auto first_chunk = column->chunk(0);
+    ASSERT_EQ(first_chunk->offset(), offset);
+
+    // write out the sliced table, read it back and compare
+    ASSERT_OK(WriteAndGetParquetInfo(sliced_table, kMinChunkSize, kMaxChunkSize, true,
+                                     param.data_page_version));
+  }
+}
+
+INSTANTIATE_TEST_SUITE_P(
+    FixedSizedTypes, TestCDCSingleRowGroup,
+    testing::Values(
+        // Boolean
+        CaseConfig{::arrow::boolean(), false, 1},
+        // Numeric
+        CaseConfig{::arrow::uint8(), false, 1}, CaseConfig{::arrow::uint16(), false, 2},
+        CaseConfig{::arrow::uint32(), false, 4}, CaseConfig{::arrow::uint64(), true, 8},
+        CaseConfig{::arrow::int8(), false, 1}, CaseConfig{::arrow::int16(), false, 2},
+        CaseConfig{::arrow::int32(), false, 4}, CaseConfig{::arrow::int64(), true, 8},
+        CaseConfig{::arrow::float16(), false, 2},
+        CaseConfig{::arrow::float32(), false, 4}, CaseConfig{::arrow::float64(), true, 8},
+        CaseConfig{::arrow::decimal128(18, 6), false, 16},
+        CaseConfig{::arrow::decimal256(40, 6), false, 32},
+        // Binary-like
+        CaseConfig{::arrow::utf8(), false, 16}, CaseConfig{::arrow::binary(), true, 16},
+        CaseConfig{::arrow::fixed_size_binary(16), true, 16},
+        // Temporal
+        CaseConfig{::arrow::date32(), false, 4},
+        CaseConfig{::arrow::time32(::arrow::TimeUnit::MILLI), true, 4},
+        CaseConfig{::arrow::time64(::arrow::TimeUnit::NANO), false, 8},
+        CaseConfig{::arrow::timestamp(::arrow::TimeUnit::NANO), true, 8},
+        CaseConfig{::arrow::duration(::arrow::TimeUnit::NANO), false, 8},
+        // Nested types
+        CaseConfig{::arrow::list(::arrow::int32()), false, 16},
+        CaseConfig{::arrow::list(::arrow::int32()), true, 18},
+        CaseConfig{::arrow::list(::arrow::utf8()), true, 18},
+        CaseConfig{::arrow::struct_({::arrow::field("f0", ::arrow::int32())}), false, 8},
+        CaseConfig{::arrow::struct_({::arrow::field("f0", ::arrow::float64())}), true,
+                   10},
+        CaseConfig{
+            ::arrow::list(::arrow::struct_({::arrow::field("f0", ::arrow::int32())})),
+            false, 16},
+        // Extension type
+        CaseConfig{::arrow::uuid(), true, 16},
+        // Use ParquetDataPageVersion::V2
+        CaseConfig{::arrow::large_binary(), false, 16, ParquetDataPageVersion::V2},
+        CaseConfig{::arrow::list(::arrow::utf8()), true, 18,
+                   ParquetDataPageVersion::V2}));
+
+class TestCDCMultipleRowGroups : public ::testing::Test {
+ protected:
+  // Column random table parts for testing
+  std::shared_ptr<DataType> dtype_;
+  std::shared_ptr<Table> part1_, part2_, part3_;
+  std::shared_ptr<Table> edit1_, edit2_, edit3_;
+
+  void SetUp() override {
+    auto constexpr kPartLength = 256 * 1024;
+    auto constexpr kEditLength = 128;
+
+    dtype_ = ::arrow::int32();
+    auto field = ::arrow::field("f0", dtype_, true);
+    auto schema = ::arrow::schema({field});
+
+    ASSERT_OK_AND_ASSIGN(part1_, GenerateTable(schema, kPartLength, 0));
+    ASSERT_OK_AND_ASSIGN(part2_, GenerateTable(schema, kPartLength, 2));
+    ASSERT_OK_AND_ASSIGN(part3_, GenerateTable(schema, kPartLength, 4));
+
+    ASSERT_OK_AND_ASSIGN(edit1_, GenerateTable(schema, kEditLength, 1));
+    ASSERT_OK_AND_ASSIGN(edit2_, GenerateTable(schema, kEditLength, 3));
+    ASSERT_OK_AND_ASSIGN(edit3_, GenerateTable(schema, kEditLength, 5));
+  }
+};
+
+TEST_F(TestCDCMultipleRowGroups, InsertOnce) {
+  auto constexpr kRowGroupLength = 128 * 1024;
+  auto constexpr kEnableDictionary = false;
+  auto constexpr kMinChunkSize = 0 * 1024;
+  auto constexpr kMaxChunkSize = 128 * 1024;
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, edit1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto inserted,
+                       ConcatAndCombine({part1_, edit1_, edit2_, part2_, part3_}));
+  ASSERT_FALSE(base->Equals(*inserted));
+  ASSERT_EQ(inserted->num_rows(), base->num_rows() + edit2_->num_rows());
+
+  ASSERT_OK_AND_ASSIGN(
+      auto base_info,
+      WriteAndGetParquetInfo(base, kMinChunkSize, kMaxChunkSize, kEnableDictionary,
+                             ParquetDataPageVersion::V1, kRowGroupLength));
+  ASSERT_OK_AND_ASSIGN(
+      auto inserted_info,
+      WriteAndGetParquetInfo(inserted, kMinChunkSize, kMaxChunkSize, kEnableDictionary,
+                             ParquetDataPageVersion::V1, kRowGroupLength));
+
+  ASSERT_EQ(base_info.size(), 7);
+  ASSERT_EQ(inserted_info.size(), 7);
+
+  ASSERT_EQ(base_info.at(0).page_lengths, inserted_info.at(0).page_lengths);
+  ASSERT_EQ(base_info.at(1).page_lengths, inserted_info.at(1).page_lengths);
+  for (size_t i = 2; i < inserted_info.size() - 1; i++) {
+    AssertPageLengthDifferences(base_info.at(i), inserted_info.at(i),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/1,
+                                /*exact_number_of_smaller_diffs=*/1, edit2_->num_rows());
+  }
+  AssertPageLengthDifferences(base_info.back(), inserted_info.back(),
+                              /*exact_number_of_equal_diffs=*/0,
+                              /*exact_number_of_larger_diffs=*/1,
+                              /*exact_number_of_smaller_diffs=*/0, edit2_->num_rows());
+}
+
+TEST_F(TestCDCMultipleRowGroups, DeleteOnce) {
+  auto constexpr kRowGroupLength = 128 * 1024;
+  auto constexpr kEnableDictionary = false;
+  auto constexpr kMinChunkSize = 0 * 1024;
+  auto constexpr kMaxChunkSize = 128 * 1024;
+
+  ASSERT_OK_AND_ASSIGN(auto base,
+                       ConcatAndCombine({part1_, edit1_, part2_, part3_, edit2_}));
+  ASSERT_OK_AND_ASSIGN(auto deleted, ConcatAndCombine({part1_, part2_, part3_, edit2_}));
+  ASSERT_FALSE(base->Equals(*deleted));
+  ASSERT_EQ(deleted->num_rows(), base->num_rows() - edit1_->num_rows());
+
+  ASSERT_OK_AND_ASSIGN(
+      auto base_info,
+      WriteAndGetParquetInfo(base, kMinChunkSize, kMaxChunkSize, kEnableDictionary,
+                             ParquetDataPageVersion::V1, kRowGroupLength));
+  ASSERT_OK_AND_ASSIGN(
+      auto deleted_info,
+      WriteAndGetParquetInfo(deleted, kMinChunkSize, kMaxChunkSize, kEnableDictionary,
+                             ParquetDataPageVersion::V1, kRowGroupLength));
+
+  ASSERT_EQ(base_info.size(), 7);
+  ASSERT_EQ(deleted_info.size(), 7);
+
+  ASSERT_EQ(base_info.at(0).page_lengths, deleted_info.at(0).page_lengths);
+  ASSERT_EQ(base_info.at(1).page_lengths, deleted_info.at(1).page_lengths);
+  for (size_t i = 2; i < deleted_info.size() - 1; i++) {
+    AssertPageLengthDifferences(base_info.at(i), deleted_info.at(i),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/1,
+                                /*exact_number_of_smaller_diffs=*/1, edit1_->num_rows());

Review Comment:
   Added comments to all these scenarios.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027909495


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}

Review Comment:
   Well, no. Making reliable assertions on them is fairly hard because the parquet page sizes reflect the encoded values and encoded levels whereas the CDC size calculation happens on the logical values. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1983884021


##########
cpp/src/parquet/chunker_internal_hashtable.h:
##########
@@ -0,0 +1,545 @@
+// Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   Renamed, added to git attributes, also renamed the generator file to `chunker_internal_codegen.py`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1983932351


##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage system (CAS)
+/// which splits the bytes stream into content defined blobs. The CAS system will
+/// calculate a unique identifier for each blob, then store the blob in a key-value store.
+/// If the same blob is encountered again, the system can refer to the hash instead of
+/// physically storing the blob again. In the example above, the CAS system would store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks.
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data
+///   Deduplication
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+/// - Git is for Data (chunk size normalization used here is described in section 6.2.1):
+///   https://www.cidrdb.org/cidr2023/papers/p43-low.pdf
+class ContentDefinedChunker {
+ public:
+  /// Create a new ContentDefinedChunker instance
+  ///
+  /// @param level_info Information about definition and repetition levels
+  /// @param size_range Min/max chunk size as pair<min_size, max_size>, the chunker will
+  ///                   attempt to uniformly distribute the chunks between these extremes.
+  /// @param norm_factor Normalization factor to center the chunk size around the average
+  ///                    size more aggressively. By increasing the normalization factor,
+  ///                    probability of finding a chunk boundary increases.

Review Comment:
   > Well, `min_size` ensures that the pages don't get too small, right?
   Yes, also skips hashing for some performance benefits.
   
   > I also don't know why it's called "normalization factor". Nothing gets normalized here.
   The [FastCDC paper](https://csyhua.github.io/csyhua/hua-atc2016.pdf) refers this "mask bit adjustment" as "normalization level". Our approach is more accurate by requiring 8 consecutive match using 8 independent gear hash tables. 
   
   By default the calculated mask attempts to have a match after every `avg_size - min_size`, and does a pretty good job, so generally the norm_factor argument shouldn't be used. I just exposed it so that the user can increase the probability of having a match (also useful for evaluation). 
   
   Tried to explain it better in the docstring:
   
   ```
     /// @param norm_factor Normalization factor to center the chunk size around the average
     ///                    size more aggressively. By increasing the normalization factor,
     ///                    probability of finding a chunk boundary increases improving the
     ///                    deduplication ratio, but also increases the number of small
     ///                    chunks resulting in small parquet data pages. The default value
     ///                    provides a good balance between deduplication ratio and
     ///                    fragmentation. Use norm_factor=1 or norm_factor=2 if a higher
     ///                    deduplication ratio is required at the expense of fragmentation,
     ///                    norm_factor>2 is typically not increasing the deduplication
     ///                    ratio.
   ```
   
   It would be better to have a less vague parameter name than `norm_factor`, but I couldn't come up with a better one just for now. I am open to suggestions. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1983571544


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,250 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_hashtable.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][bytes[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);

Review Comment:
   We need to distinguish between different values like `first-value` and `second-value` to observe that there is a change. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1983890061


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,250 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_hashtable.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][bytes[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1985249871


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, uint64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal128Builder, ::arrow::Decimal128(val % max_value))
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal256Builder, ::arrow::Decimal256(val % max_value))
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      GENERATE_CASE_BODY(::arrow::FixedSizeBinaryBuilder,
+                         std::string("bin_") + std::to_string(val).substr(0, size - 4))
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, uint64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(const std::shared_ptr<Table>& table,
+                                                   uint64_t min_chunk_size,
+                                                   uint64_t max_chunk_size,

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086165448


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,412 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+namespace {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_level parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_level Normalization level (default 0)
+// @return The mask used to compare against the rolling hash
+uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size, int norm_level) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be non-negative");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes; also divide by the number of gearhash tables to have a
+  // a more gaussian-like distribution
+  int64_t target_size = (avg_chunk_size - min_chunk_size) / kNumGearhashTables;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // a user defined `norm_level` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_level we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_level is 0 by
+  // default
+  int effective_bits = mask_bits - norm_level;
+
+  if (effective_bits < 1 || effective_bits > 63) {
+    throw ParquetException(
+        "The number of bits in the CDC mask must be between 1 and 63, got " +
+        std::to_string(effective_bits));
+  } else {
+    // create the mask by setting the top bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+}  // namespace
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int norm_level)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_level)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the

Review Comment:
   The goal is not only informative since two versions of the same dataset should be written with the same chunking parametrization to have proper deduplication on the storage level. 
   
   Regarding the max page size, CDC doesn't override it, that is still a hard upper limit on the serialized page size.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086220901


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,412 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+namespace {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_level parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_level Normalization level (default 0)
+// @return The mask used to compare against the rolling hash
+uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size, int norm_level) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be non-negative");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes; also divide by the number of gearhash tables to have a
+  // a more gaussian-like distribution
+  int64_t target_size = (avg_chunk_size - min_chunk_size) / kNumGearhashTables;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // a user defined `norm_level` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_level we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_level is 0 by
+  // default
+  int effective_bits = mask_bits - norm_level;
+
+  if (effective_bits < 1 || effective_bits > 63) {
+    throw ParquetException(
+        "The number of bits in the CDC mask must be between 1 and 63, got " +
+        std::to_string(effective_bits));
+  } else {
+    // create the mask by setting the top bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+}  // namespace
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int norm_level)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_level)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the

Review Comment:
   `WriterProperties` contains most of the controllable properties, ideally we should be able to serialize those settings in the KV metadata. Though I am not sure about the serialization format since JSON is not available. 
   
   Another example for this use case could be storing page indexes for existing parquet files with keeping the exact same row-group/page layout.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["mapleFU (via GitHub)" <gi...@apache.org>]

mapleFU commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2013586539


##########
cpp/src/parquet/properties.h:
##########
@@ -260,7 +268,9 @@ class PARQUET_EXPORT WriterProperties {
           created_by_(DEFAULT_CREATED_BY),
           store_decimal_as_integer_(false),
           page_checksum_enabled_(false),
-          size_statistics_level_(DEFAULT_SIZE_STATISTICS_LEVEL) {}
+          size_statistics_level_(DEFAULT_SIZE_STATISTICS_LEVEL),
+          content_defined_chunking_enabled_(false),

Review Comment:
   I'm think of would a `optional<CDCOptions>` better here?



##########
cpp/src/parquet/properties.h:
##########
@@ -260,7 +268,9 @@ class PARQUET_EXPORT WriterProperties {
           created_by_(DEFAULT_CREATED_BY),
           store_decimal_as_integer_(false),
           page_checksum_enabled_(false),
-          size_statistics_level_(DEFAULT_SIZE_STATISTICS_LEVEL) {}
+          size_statistics_level_(DEFAULT_SIZE_STATISTICS_LEVEL),
+          content_defined_chunking_enabled_(false),

Review Comment:
   Current impl also LGTM



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["mapleFU (via GitHub)" <gi...@apache.org>]

mapleFU commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2766409937

   I'm a little busy today and will take a careful round tomorrow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2014046863


##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+#include "parquet/properties.h"
+
+namespace parquet::internal {
+
+class TestPina;
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage system (CAS)
+/// which splits the bytes stream into content defined blobs. The CAS system will
+/// calculate a unique identifier for each blob, then store the blob in a key-value store.
+/// If the same blob is encountered again, the system can refer to the hash instead of
+/// physically storing the blob again. In the example above, the CAS system would store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks.
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data
+///   Deduplication
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+/// - Git is for Data (chunk size normalization used here is described in section 6.2.1):
+///   https://www.cidrdb.org/cidr2023/papers/p43-low.pdf
+class PARQUET_EXPORT ContentDefinedChunker {
+ public:
+  /// Create a new ContentDefinedChunker instance
+  ///
+  /// @param level_info Information about definition and repetition levels
+  /// @param min_chunk_size Minimum chunk size in bytes
+  ///   The rolling hash will not be updated until this size is reached for each chunk.
+  ///   Note that all data sent through the hash function is counted towards the chunk
+  ///   size, including definition and repetition levels if present.
+  /// @param max_chunk_size Maximum chunk size in bytes
+  ///   The chunker creates a new chunk whenever the chunk size exceeds this value. The
+  ///   chunk size distribution approximates a normal distribution between min_chunk_size
+  ///   and max_chunk_size. Note that the parquet writer has a related `data_pagesize`
+  //    property that controls the maximum size of a parquet data page after encoding.
+  ///   While setting `data_pagesize` to a smaller value than `max_chunk_size` doesn't
+  ///   affect the chunking effectiveness, it results in more small parquet data pages.
+  /// @param norm_factor Normalization factor to center the chunk size around the average
+  ///   size more aggressively, default 0.
+  ///   Increasing the normalization factor increases the probability of finding a chunk
+  ///   boundary, improving the deduplication ratio, but also increases the number of
+  ///   small chunks resulting in many small parquet data pages. The default value
+  ///   provides a good balance between deduplication ratio and fragmentation.
+  ///   Use norm_factor=1 or norm_factor=2 to reach a higher deduplication ratio at the
+  ///   expense of fragmentation.
+  ContentDefinedChunker(const LevelInfo& level_info, int64_t min_chunk_size,
+                        int64_t max_chunk_size, int8_t norm_factor = 0);
+  ~ContentDefinedChunker();
+
+  /// Get the chunk boundaries for the given column data

Review Comment:
   It cannot return an empty `vector<Chunk>`, I added debug checks to ensure these constraints. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1992083577


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             int64_t min_size, int64_t max_size,
+                                             int8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(min_size),
+      max_size_(max_size),
+      hash_mask_(GetMask(min_size, max_size, norm_factor)) {
+  if (min_size_ < 0) {
+    throw ParquetException("min_size must be non-negative");
+  }
+  if (max_size_ < 0) {
+    throw ParquetException("max_size must be non-negative");
+  }
+  if (min_size_ > max_size_) {
+    throw ParquetException("min_size must be less than or equal to max_size");
+  }
+}
+
+void ContentDefinedChunker::Roll(const bool value) {
+  if (chunk_size_++ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+  has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+}
+
+template <int ByteWidth>
+void ContentDefinedChunker::Roll(const uint8_t* value) {
+  chunk_size_ += ByteWidth;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (size_t i = 0; i < ByteWidth; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T* value) {
+  return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+}
+
+void ContentDefinedChunker::Roll(const uint8_t* value, int64_t length) {
+  chunk_size_ += length;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (auto i = 0; i < length; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, note that we don't reset the
+    // rolling hash state here, so the next NeedNewChunk() call will continue from the
+    // current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename RollFunc>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const RollFunc& RollValue) {
+  std::vector<Chunk> chunks;
+  int64_t offset;
+  int64_t prev_offset = 0;
+  int64_t prev_value_offset = 0;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    for (offset = 0; offset < num_levels; ++offset) {
+      RollValue(offset);
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int16_t def_level;
+    for (int64_t offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+
+      Roll(&def_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(offset);
+      }
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else {
+    // nested data with nulls
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t value_offset = 0;
+
+    for (offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+      rep_level = rep_levels[offset];
+
+      Roll(&def_level);
+      Roll(&rep_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(value_offset);
+      }
+
+      if ((rep_level == 0) && NeedNewChunk()) {
+        // if we are at a record boundary and need a new chunk, we create a new chunk
+        auto levels_to_write = offset - prev_offset;
+        if (levels_to_write > 0) {
+          chunks.emplace_back(prev_offset, prev_value_offset, levels_to_write);
+          prev_offset = offset;
+          prev_value_offset = value_offset;
+        }
+      }
+      if (def_level >= level_info_.repeated_ancestor_def_level) {
+        // we only increment the value offset if we have a leaf value
+        ++value_offset;
+      }
+    }
+  }
+
+  // add the last chunk if we have any levels left
+  if (prev_offset < num_levels) {
+    chunks.emplace_back(prev_offset, prev_value_offset, num_levels - prev_offset);
+  }
+  return chunks;
+}
+
+#define FIXED_WIDTH_CASE(ByteWidth)                                       \
+  {                                                                       \
+    const auto raw_values = values.data()->GetValues<uint8_t>(1);         \
+    return Calculate(def_levels, rep_levels, num_levels, [&](int64_t i) { \
+      return Roll<ByteWidth>(raw_values + i * ByteWidth);                 \
+    });                                                                   \
+  }
+
+#define BINARY_LIKE_CASE(ArrayType)                                       \
+  {                                                                       \
+    const auto& array = static_cast<const ArrayType&>(values);            \
+    const uint8_t* value;                                                 \
+    ArrayType::offset_type length;                                        \
+    return Calculate(def_levels, rep_levels, num_levels, [&](int64_t i) { \
+      value = array.GetValue(i, &length);                                 \
+      Roll(value, length);                                                \
+    });                                                                   \
+  }
+
+const std::vector<Chunk> ContentDefinedChunker::GetBoundaries(
+    const int16_t* def_levels, const int16_t* rep_levels, int64_t num_levels,
+    const ::arrow::Array& values) {
+  auto type_id = values.type()->id();
+  switch (type_id) {

Review Comment:
   SGTM, updating.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1993014152


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             int64_t min_size, int64_t max_size,
+                                             int8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(min_size),
+      max_size_(max_size),
+      hash_mask_(GetMask(min_size, max_size, norm_factor)) {
+  if (min_size_ < 0) {
+    throw ParquetException("min_size must be non-negative");
+  }
+  if (max_size_ < 0) {
+    throw ParquetException("max_size must be non-negative");
+  }
+  if (min_size_ > max_size_) {
+    throw ParquetException("min_size must be less than or equal to max_size");
+  }
+}
+
+void ContentDefinedChunker::Roll(const bool value) {
+  if (chunk_size_++ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+  has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+}
+
+template <int ByteWidth>
+void ContentDefinedChunker::Roll(const uint8_t* value) {
+  chunk_size_ += ByteWidth;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (size_t i = 0; i < ByteWidth; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T* value) {
+  return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+}
+
+void ContentDefinedChunker::Roll(const uint8_t* value, int64_t length) {
+  chunk_size_ += length;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (auto i = 0; i < length; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, note that we don't reset the
+    // rolling hash state here, so the next NeedNewChunk() call will continue from the
+    // current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename RollFunc>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,

Review Comment:
   Removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025227688


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();
+    auto min_length = ElementCount(min_chunk_size, byte_width, nullable);
+    auto max_length = ElementCount(max_chunk_size, byte_width, nullable);
+    AssertAllBetween(base_info.page_lengths, min_length, max_length);
+    AssertAllBetween(modified_info.page_lengths, min_length, max_length);
+  } else if (::arrow::is_base_binary_like(dtype->id()) && !enable_dictionary) {
+    AssertAllBetween(base_info.page_sizes, min_chunk_size, max_chunk_size);
+    AssertAllBetween(modified_info.page_sizes, min_chunk_size, max_chunk_size);
+  }
+}
+
+constexpr int64_t kMinChunkSize = 8 * 1024;
+constexpr int64_t kMaxChunkSize = 32 * 1024;
+constexpr int64_t kPartSize = 128 * 1024;
+constexpr int64_t kEditSize = 128;
+
+struct CaseConfig {
+  // Arrow data type to generate the testing data for
+  std::shared_ptr<::arrow::DataType> dtype;
+  // Whether the data type is nullable
+  bool is_nullable;
+  // Approximate number of bytes per record to calculate the number of elements to
+  // generate
+  size_t bytes_per_record;
+  // Data page version to use
+  ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1;
+};
+
+// Define PrintTo for MyStruct
+void PrintTo(const CaseConfig& param, std::ostream* os) {
+  *os << "{ " << param.dtype->ToString();
+  if (param.is_nullable) {
+    *os << " nullable";
+  }
+  *os << " }";
+}
+
+class TestCDC : public ::testing::Test {
+ public:
+  uint64_t GetMask(const ContentDefinedChunker& cdc) const { return cdc.GetMask(); }
+};
+
+TEST_F(TestCDC, RollingHashMaskCalculation) {
+  auto le = LevelInfo();
+  auto min_size = 256 * 1024;
+  auto max_size = 1024 * 1024;
+
+  auto cdc0 = ContentDefinedChunker(le, min_size, max_size, 0);
+  ASSERT_EQ(GetMask(cdc0), 0xFFFE000000000000);
+
+  auto cdc1 = ContentDefinedChunker(le, min_size, max_size, 1);
+  ASSERT_EQ(GetMask(cdc1), 0xFFFC000000000000);
+
+  auto cdc2 = ContentDefinedChunker(le, min_size, max_size, 2);
+  ASSERT_EQ(GetMask(cdc2), 0xFFF8000000000000);
+
+  auto cdc3 = ContentDefinedChunker(le, min_size, max_size, 3);
+  ASSERT_EQ(GetMask(cdc3), 0xFFF0000000000000);
+
+  auto cdc4 = ContentDefinedChunker(le, min_size, max_size, -1);
+  ASSERT_EQ(GetMask(cdc4), 0xFFFF000000000000);
+
+  // this is the smallest possible mask always matching, by using 8 hashtables
+  // we are going to have a match every 8 bytes; this is an unrealistic case
+  // but checking for the correctness of the mask calculation
+  auto cdc5 = ContentDefinedChunker(le, 0, 16, 0);
+  ASSERT_EQ(GetMask(cdc5), 0x0000000000000000);
+
+  auto cdc6 = ContentDefinedChunker(le, 0, 32, 1);
+  ASSERT_EQ(GetMask(cdc6), 0x0000000000000000);
+
+  auto cdc7 = ContentDefinedChunker(le, 0, 16, -1);
+  ASSERT_EQ(GetMask(cdc7), 0x8000000000000000);
+
+  // another unrealistic case, checking for the validation
+  auto cdc8 = ContentDefinedChunker(le, 128, 384, -60);
+  ASSERT_EQ(GetMask(cdc8), 0xFFFFFFFFFFFFFFFF);
+}
+
+TEST_F(TestCDC, WriteSingleColumnParquetFile) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  std::vector<int32_t> numbers = {1, 2, 3, 4, 5};
+  std::vector<uint8_t> valid_bits = {1, 0, 1, 0, 1};
+  EXPECT_THROW(
+      int_column_writer.WriteBatch(numbers.size(), nullptr, nullptr, numbers.data()),
+      ParquetException);
+  EXPECT_THROW(int_column_writer.WriteBatchSpaced(numbers.size(), nullptr, nullptr,
+                                                  valid_bits.data(), 0, numbers.data()),
+               ParquetException);
+}
+
+TEST_F(TestCDC, LastChunkDoesntTriggerAddDataPage) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()
+                   ->content_defined_chunking_options(kMinChunkSize, kMaxChunkSize, 0)
+                   ->disable_dictionary()
+                   ->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  ASSERT_OK_AND_ASSIGN(auto array, ::arrow::gen::Step()->Generate(8000));
+  auto arrow_props = default_arrow_writer_properties();
+  auto arrow_ctx = ArrowWriteContext(default_memory_pool(), arrow_props.get());
+
+  // Calling WriteArrow twice, we expect that the first call doesn't add a new data page
+  // at the end allowing subsequent calls to append to the same page
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+
+  int_column_writer.Close();
+  writer->Close();
+  ASSERT_OK_AND_ASSIGN(auto buffer, sink->Finish());
+
+  auto info = GetColumnParquetInfo(buffer);
+  ASSERT_EQ(info.size(), 1);
+
+  // AssertAllBetween allow the last chunk size to be smaller than the min_chunk_size
+  AssertAllBetween(info[0].page_sizes, kMinChunkSize, kMaxChunkSize);
+  AssertAllBetween(info[0].page_lengths, 3000, 5000);
+}
+
+TEST_F(TestCDC, ChunkSizeParameterValidation) {
+  // Test that constructor validates min/max chunk size parameters
+  auto li = LevelInfo();
+
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 256 * 1024, 1024 * 1024));
+
+  // with norm_factor=0 the difference between min and max chunk size must be
+  // at least 16
+  ASSERT_THROW(ContentDefinedChunker(li, 0, -1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1024, 512), ParquetException);
+
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16));
+  ASSERT_THROW(ContentDefinedChunker(li, -16, -16), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 16, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 32, 32), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 32, 48));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1024 * 1024, 2 * 1024 * 1024));
+  ASSERT_NO_THROW(
+      ContentDefinedChunker(li, 1024 * 1024 * 1024L, 2LL * 1024 * 1024 * 1024L));
+
+  // with norm_factor=1 the difference between min and max chunk size must be
+  // at least 64
+  ASSERT_THROW(ContentDefinedChunker(li, 1, -1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 32, 1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1, 33, 1));
+
+  // with norm_factor=2 the difference between min and max chunk size must be
+  // at least 128
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 63, 2), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 64, 2));
+
+  // with norm_factor=-1 the difference between min and max chunk size must be
+  // at least 8
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 7, -1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 8, -1));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16, -1));
+
+  // test the norm_factor extremes
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0, -68), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 1, -67));
+  ASSERT_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 59),
+               ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 58));
+}
+
+class TestCDCSingleRowGroup : public ::testing::TestWithParam<CaseConfig> {
+ protected:
+  // Column random table parts for testing
+  std::shared_ptr<Field> field_;
+  std::shared_ptr<Table> part1_, part2_, part3_, part4_, part5_, part6_, part7_;
+
+  void SetUp() override {
+    const auto& param = GetParam();
+    auto field_ = ::arrow::field("f0", param.dtype, param.is_nullable);
+    auto schema = ::arrow::schema({field_});
+
+    auto part_length = kPartSize / param.bytes_per_record;
+    auto edit_length = kEditSize / param.bytes_per_record;
+    ASSERT_OK_AND_ASSIGN(part1_, GenerateTable(schema, part_length, 0));
+    ASSERT_OK_AND_ASSIGN(part2_, GenerateTable(schema, edit_length, 1));
+    ASSERT_OK_AND_ASSIGN(part3_, GenerateTable(schema, part_length, part_length));
+    ASSERT_OK_AND_ASSIGN(part4_, GenerateTable(schema, edit_length, 2));
+    ASSERT_OK_AND_ASSIGN(part5_, GenerateTable(schema, part_length, 2 * part_length));
+    ASSERT_OK_AND_ASSIGN(part6_, GenerateTable(schema, edit_length, 3));
+    ASSERT_OK_AND_ASSIGN(part7_, GenerateTable(schema, edit_length, 4));
+  }
+};
+
+TEST_P(TestCDCSingleRowGroup, DeleteOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/0,
+                                /*exact_number_of_smaller_diffs=*/1, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, DeleteTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part3_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/0,
+                                /*exact_number_of_smaller_diffs=*/2, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, UpdateOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part4_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*max_number_of_equal_diffs=*/1);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, UpdateTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified,
+                       ConcatAndCombine({part1_, part6_, part3_, part7_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*max_number_of_equal_diffs=*/2);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, InsertOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/1,
+                                /*exact_number_of_smaller_diffs=*/0, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, InsertTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part3_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/2,
+                                /*exact_number_of_smaller_diffs=*/0, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, Append) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part2_, part3_, part4_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto original_page_lengths = base_info.front().page_lengths;
+    auto modified_page_lengths = modified_info.front().page_lengths;
+    ASSERT_GE(original_page_lengths.size(), modified_page_lengths.size());
+    for (size_t i = 0; i < original_page_lengths.size() - 1; i++) {
+      ASSERT_EQ(original_page_lengths[i], modified_page_lengths[i]);
+    }
+    ASSERT_GT(modified_page_lengths.back(), original_page_lengths.back());
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, EmptyTable) {
+  const auto& param = GetParam();
+
+  auto schema = ::arrow::schema({::arrow::field("f0", param.dtype, param.is_nullable)});
+  ASSERT_OK_AND_ASSIGN(auto empty_table, GenerateTable(schema, 0, 0));
+  ASSERT_EQ(empty_table->num_rows(), 0);
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(
+        auto result, WriteAndGetParquetInfo(empty_table, kMinChunkSize, kMaxChunkSize,
+                                            enable_dictionary, param.data_page_version));
+
+    // An empty table should result in no data pages
+    ASSERT_EQ(result.size(), 1);
+    ASSERT_TRUE(result.front().page_lengths.empty());
+    ASSERT_TRUE(result.front().page_sizes.empty());
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, ArrayOffsets) {
+  const auto& param = GetParam();
+  ASSERT_OK_AND_ASSIGN(auto table, ConcatAndCombine({part1_, part2_, part3_}));
+
+  for (auto offset : {0, 512, 1024}) {
+    auto sliced_table = table->Slice(offset);
+
+    // assert that the first column has a non-zero offset
+    auto column = sliced_table->column(0);
+    auto first_chunk = column->chunk(0);
+    ASSERT_EQ(first_chunk->offset(), offset);
+
+    // write out the sliced table, read it back and compare
+    ASSERT_OK(WriteAndGetParquetInfo(sliced_table, kMinChunkSize, kMaxChunkSize, true,
+                                     param.data_page_version));
+  }
+}
+
+INSTANTIATE_TEST_SUITE_P(
+    FixedSizedTypes, TestCDCSingleRowGroup,
+    testing::Values(
+        // Boolean
+        CaseConfig{::arrow::boolean(), false, 1},
+        // Numeric
+        CaseConfig{::arrow::uint8(), false, 1}, CaseConfig{::arrow::uint16(), false, 2},
+        CaseConfig{::arrow::uint32(), false, 4}, CaseConfig{::arrow::uint64(), true, 8},
+        CaseConfig{::arrow::int8(), false, 1}, CaseConfig{::arrow::int16(), false, 2},
+        CaseConfig{::arrow::int32(), false, 4}, CaseConfig{::arrow::int64(), true, 8},
+        CaseConfig{::arrow::float16(), false, 2},
+        CaseConfig{::arrow::float32(), false, 4}, CaseConfig{::arrow::float64(), true, 8},
+        CaseConfig{::arrow::decimal128(18, 6), false, 16},
+        CaseConfig{::arrow::decimal256(40, 6), false, 32},
+        // Binary-like
+        CaseConfig{::arrow::utf8(), false, 16}, CaseConfig{::arrow::binary(), true, 16},
+        CaseConfig{::arrow::fixed_size_binary(16), true, 16},
+        // Temporal
+        CaseConfig{::arrow::date32(), false, 4},
+        CaseConfig{::arrow::time32(::arrow::TimeUnit::MILLI), true, 4},
+        CaseConfig{::arrow::time64(::arrow::TimeUnit::NANO), false, 8},
+        CaseConfig{::arrow::timestamp(::arrow::TimeUnit::NANO), true, 8},
+        CaseConfig{::arrow::duration(::arrow::TimeUnit::NANO), false, 8},
+        // Nested types
+        CaseConfig{::arrow::list(::arrow::int32()), false, 16},
+        CaseConfig{::arrow::list(::arrow::int32()), true, 18},
+        CaseConfig{::arrow::list(::arrow::utf8()), true, 18},
+        CaseConfig{::arrow::struct_({::arrow::field("f0", ::arrow::int32())}), false, 8},
+        CaseConfig{::arrow::struct_({::arrow::field("f0", ::arrow::float64())}), true,
+                   10},
+        CaseConfig{
+            ::arrow::list(::arrow::struct_({::arrow::field("f0", ::arrow::int32())})),
+            false, 16},
+        // Extension type
+        CaseConfig{::arrow::uuid(), true, 16},
+        // Use ParquetDataPageVersion::V2
+        CaseConfig{::arrow::large_binary(), false, 16, ParquetDataPageVersion::V2},
+        CaseConfig{::arrow::list(::arrow::utf8()), true, 18,
+                   ParquetDataPageVersion::V2}));
+
+class TestCDCMultipleRowGroups : public ::testing::Test {
+ protected:
+  // Column random table parts for testing
+  std::shared_ptr<DataType> dtype_;
+  std::shared_ptr<Table> part1_, part2_, part3_;
+  std::shared_ptr<Table> edit1_, edit2_, edit3_;
+
+  void SetUp() override {
+    auto constexpr kPartLength = 256 * 1024;
+    auto constexpr kEditLength = 128;

Review Comment:
   Good idea, updated. Yes, I'm using that naming convention.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086966277


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1689 @@
+// Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   Updated, now running a subset of test cases for asan/valgrind.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086148223


##########
docs/source/python/parquet.rst:
##########
@@ -782,3 +782,63 @@ file decryption properties) is optional and it includes the following options:
 
 * ``cache_lifetime``, the lifetime of cached entities (key encryption keys, local
   wrapping keys, KMS client objects) represented as a ``datetime.timedelta``.
+
+
+Content-Defined Chunking
+------------------------
+
+.. note::
+   This feature is experimental and may change in future releases.
+
+PyArrow introduces an experimental feature for optimizing Parquet files for content
+addressable storage (CAS) systems using content-defined chunking (CDC). This feature
+enables efficient deduplication of data across files, improving network transfers and
+storage efficiency.
+
+When enabled, data pages are written according to content-defined chunk boundaries,
+determined by a rolling hash algorithm that identifies chunk boundaries based on the
+actual content of the data. When data in a column is modified (e.g., inserted, deleted,
+or updated), this approach minimizes the number of changed data pages.
+
+The feature can be enabled by setting the ``use_content_defined_chunking`` parameter in
+the Parquet writer. It accepts either a boolean or a dictionary for configuration:
+
+- ``True``: Uses the default configuration with:
+   - Minimum chunk size: 256 KiB
+   - Maximum chunk size: 1024 KiB
+   - Normalization level: 0
+
+- ``dict``: Allows customization of the chunking parameters:
+   - ``min_chunk_size``: Minimum chunk size in bytes (default: 256 KiB).
+   - ``max_chunk_size``: Maximum chunk size in bytes (default: 1024 KiB).
+   - ``norm_level``: Normalization level to adjust chunk size distribution (default: 0).
+
+Note that the chunk size is calculated on the logical values before applying any encoding
+or compression. The actual size of the data pages may vary based on the encoding and
+compression used.
+
+.. note::
+   Ensure that Parquet write options remain consistent across writes and files.
+   Using different write options (like compression, encoding, or row group size)
+   for different files may prevent proper deduplication and lead to suboptimal
+   storage efficiency.

Review Comment:
   ```suggestion
   .. note::
      To make the most of this feature, you should ensure that Parquet write options
      remain consistent across writes and files.
      Using different write options (like compression, encoding, or row group size)
      for different files may prevent proper deduplication and lead to suboptimal
      storage efficiency.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2732302948

   Revision: fc304686de3584dd0856fee3c8dfa8d2f4fa2107
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-6da1e25b80](https://github.com/ursacomputing/crossbow/branches/all?query=actions-6da1e25b80)
   
   |Task|Status|
   |----|------|
   |wheel-macos-monterey-cp310-cp310-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-macos-monterey-cp310-cp310-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837448/job/38950284479)|
   |wheel-macos-monterey-cp310-cp310-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-macos-monterey-cp310-cp310-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837468/job/38950284493)|
   |wheel-macos-monterey-cp311-cp311-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-macos-monterey-cp311-cp311-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919836757/job/38950281523)|
   |wheel-macos-monterey-cp311-cp311-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-macos-monterey-cp311-cp311-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837369/job/38950284059)|
   |wheel-macos-monterey-cp312-cp312-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-macos-monterey-cp312-cp312-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837571/job/38950285847)|
   |wheel-macos-monterey-cp312-cp312-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-macos-monterey-cp312-cp312-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837183/job/38950283479)|
   |wheel-macos-monterey-cp313-cp313-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-macos-monterey-cp313-cp313-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919836679/job/38950281524)|
   |wheel-macos-monterey-cp313-cp313-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-macos-monterey-cp313-cp313-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837788/job/38950287778)|
   |wheel-macos-monterey-cp313-cp313t-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-macos-monterey-cp313-cp313t-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837962/job/38950289029)|
   |wheel-macos-monterey-cp313-cp313t-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-macos-monterey-cp313-cp313t-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/13919838001/job/38950289369)|
   |wheel-macos-monterey-cp39-cp39-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-macos-monterey-cp39-cp39-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837674/job/38950286610)|
   |wheel-macos-monterey-cp39-cp39-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-macos-monterey-cp39-cp39-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837533/job/38950285243)|
   |wheel-manylinux-2-28-cp310-cp310-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2-28-cp310-cp310-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837020/job/38950282936)|
   |wheel-manylinux-2-28-cp310-cp310-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2-28-cp310-cp310-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837541/job/38950285304)|
   |wheel-manylinux-2-28-cp311-cp311-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2-28-cp311-cp311-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837502/job/38950284896)|
   |wheel-manylinux-2-28-cp311-cp311-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2-28-cp311-cp311-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/13919838055/job/38950290276)|
   |wheel-manylinux-2-28-cp312-cp312-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2-28-cp312-cp312-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837035/job/38950282943)|
   |wheel-manylinux-2-28-cp312-cp312-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2-28-cp312-cp312-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837546/job/38950285436)|
   |wheel-manylinux-2-28-cp313-cp313-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2-28-cp313-cp313-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919838086/job/38950289682)|
   |wheel-manylinux-2-28-cp313-cp313-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2-28-cp313-cp313-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837552/job/38950285355)|
   |wheel-manylinux-2-28-cp313-cp313t-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2-28-cp313-cp313t-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837975/job/38950289355)|
   |wheel-manylinux-2-28-cp313-cp313t-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2-28-cp313-cp313t-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/13919836943/job/38950282388)|
   |wheel-manylinux-2-28-cp39-cp39-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2-28-cp39-cp39-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837469/job/38950284533)|
   |wheel-manylinux-2-28-cp39-cp39-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2-28-cp39-cp39-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/13919836948/job/38950282650)|
   |wheel-manylinux-2014-cp310-cp310-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2014-cp310-cp310-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919836890/job/38950282111)|
   |wheel-manylinux-2014-cp310-cp310-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2014-cp310-cp310-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837265/job/38950283388)|
   |wheel-manylinux-2014-cp311-cp311-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2014-cp311-cp311-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919836687/job/38950281463)|
   |wheel-manylinux-2014-cp311-cp311-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2014-cp311-cp311-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837956/job/38950288629)|
   |wheel-manylinux-2014-cp312-cp312-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2014-cp312-cp312-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837542/job/38950284957)|
   |wheel-manylinux-2014-cp312-cp312-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2014-cp312-cp312-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837484/job/38950284921)|
   |wheel-manylinux-2014-cp313-cp313-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2014-cp313-cp313-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837827/job/38950287807)|
   |wheel-manylinux-2014-cp313-cp313-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2014-cp313-cp313-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/13919838139/job/38950289961)|
   |wheel-manylinux-2014-cp313-cp313t-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2014-cp313-cp313t-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837173/job/38950283392)|
   |wheel-manylinux-2014-cp313-cp313t-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2014-cp313-cp313t-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/13919836900/job/38950282265)|
   |wheel-manylinux-2014-cp39-cp39-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2014-cp39-cp39-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919836939/job/38950282281)|
   |wheel-manylinux-2014-cp39-cp39-arm64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-manylinux-2014-cp39-cp39-arm64)](https://github.com/ursacomputing/crossbow/actions/runs/13919838229/job/38950290202)|
   |wheel-windows-cp310-cp310-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-windows-cp310-cp310-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837659/job/38950286077)|
   |wheel-windows-cp311-cp311-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-windows-cp311-cp311-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837311/job/38950283819)|
   |wheel-windows-cp312-cp312-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-windows-cp312-cp312-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837884/job/38950288234)|
   |wheel-windows-cp313-cp313-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-windows-cp313-cp313-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837951/job/38950288565)|
   |wheel-windows-cp313-cp313t-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-windows-cp313-cp313t-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837938/job/38950288315)|
   |wheel-windows-cp39-cp39-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-6da1e25b80-github-wheel-windows-cp39-cp39-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/13919837534/job/38950285044)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1999438198


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, uint64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal128Builder, ::arrow::Decimal128(val % max_value))
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal256Builder, ::arrow::Decimal256(val % max_value))
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      GENERATE_CASE_BODY(::arrow::FixedSizeBinaryBuilder,
+                         std::string("bin_") + std::to_string(val).substr(0, size - 4))
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, uint64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(const std::shared_ptr<Table>& table,
+                                                   uint64_t min_chunk_size,
+                                                   uint64_t max_chunk_size,
+                                                   bool enable_dictionary = false,
+
+                                                   int64_t row_group_size = 1024 * 1024) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_cdc()
+      ->cdc_size_range(min_chunk_size, max_chunk_size)
+      ->cdc_norm_factor(0);

Review Comment:
   Now there are two methods:
   - `enable_content_defined_chunking()` to toggle the chunker
   - `content_defined_chunking_options(min_size, max_size, norm_factor=0)` to set the options
   
   In the future I would like to generalize the chunking mechanism a little bit more, where we could use a `ColumnChunkingProperties/Options` struct to define the details, so subject to a refactor.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025234093


##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <vector>
+
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which is contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage (CAS) system
+/// which splits the bytes stream into content defined blobs. The CAS system will
+/// calculate a unique identifier for each blob, then store the blob in a key-value store.
+/// If the same blob is encountered again, the system can refer to the hash instead of
+/// physically storing the blob again. In the example above, the CAS system would store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks.
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data
+///   Deduplication
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+/// - Git is for Data (chunk size normalization used here is described in section 6.2.1):
+///   https://www.cidrdb.org/cidr2023/papers/p43-low.pdf
+class PARQUET_EXPORT ContentDefinedChunker {
+ public:
+  /// Create a new ContentDefinedChunker instance
+  ///
+  /// @param level_info Information about definition and repetition levels
+  /// @param min_chunk_size Minimum chunk size in bytes
+  ///   The rolling hash will not be updated until this size is reached for each chunk.
+  ///   Note that all data sent through the hash function is counted towards the chunk
+  ///   size, including definition and repetition levels if present.
+  /// @param max_chunk_size Maximum chunk size in bytes
+  ///   The chunker creates a new chunk whenever the chunk size exceeds this value. The
+  ///   chunk size distribution approximates a normal distribution between min_chunk_size
+  ///   and max_chunk_size. Note that the parquet writer has a related `data_pagesize`
+  //    property that controls the maximum size of a parquet data page after encoding.
+  ///   While setting `data_pagesize` to a smaller value than `max_chunk_size` doesn't
+  ///   affect the chunking effectiveness, it results in more small parquet data pages.
+  /// @param norm_factor Normalization factor to center the chunk size around the average
+  ///   size more aggressively, default 0.
+  ///   Increasing the normalization factor increases the probability of finding a chunk
+  ///   boundary, improving the deduplication ratio, but also increases the number of
+  ///   small chunks resulting in many small parquet data pages. The default value
+  ///   provides a good balance between deduplication ratio and fragmentation.
+  ///   Use norm_factor=1 or norm_factor=2 to reach a higher deduplication ratio at the
+  ///   expense of fragmentation.
+  ContentDefinedChunker(const LevelInfo& level_info, int64_t min_chunk_size,
+                        int64_t max_chunk_size, int8_t norm_factor = 0);
+  ~ContentDefinedChunker();
+
+  /// Get the chunk boundaries for the given column data
+  ///
+  /// @param def_levels Definition levels
+  /// @param rep_levels Repetition levels
+  /// @param num_levels Number of levels
+  /// @param values Column values as an Arrow array
+  /// @return Vector of Chunk objects representing the chunk boundaries
+  std::vector<Chunk> GetChunks(const int16_t* def_levels, const int16_t* rep_levels,
+                               int64_t num_levels, const ::arrow::Array& values);
+
+ private:
+  uint64_t GetMask() const;

Review Comment:
   Renamed it to `GetRollingHashMask` and also added a comment, it is used for testing the mask calculation. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025153238


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025034990


##########
python/pyarrow/parquet/core.py:
##########
@@ -889,6 +889,39 @@ def _sanitize_table(table, new_schema, flavor):
     - fixed_len_byte_array: for precision > 18.
 
     As a consequence, decimal columns stored in integer types are more compact.
+use_content_defined_chunking : bool or dict, default False
+    Optimize parquet files for content addressable storage (CAS) systems by writing

Review Comment:
   I mention that it is experimental a couple of lines lower. Do we have a standard to mark params as experimental?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025174532


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {

Review Comment:
   Updated.



##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {

Review Comment:
   Updated to int64_t.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025164533


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());

Review Comment:
   ```
   In file included from /Users/kszucs/Workspace/arrow/cpp/src/arrow/type.h:34:
   /Users/kszucs/Workspace/arrow/cpp/src/arrow/util/checked_cast.h:38:10: error: 'parquet::Page' is not polymorphic
      38 |   return dynamic_cast<OutputType>(value);
         |          ^                        ~~~~~
   /Users/kszucs/Workspace/arrow/cpp/src/parquet/chunker_internal_test.cc:339:26: note: in instantiation of function template specialization 'arrow::internal::checked_cast<parquet::DataPage *, parquet::Page *>' requested here
     339 |         auto data_page = checked_cast<DataPage*>(page.get());
     ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2017639249


##########
cpp/src/parquet/schema.cc:
##########
@@ -25,6 +25,7 @@
 #include <utility>
 
 #include "arrow/util/logging.h"
+#include "arrow/util/windows_fixup.h"  // for OPTIONAL

Review Comment:
   I had to add the windows mixup here to overcome 
   
   ```
   In file included from D:/a/arrow/arrow/build/cpp/src/parquet/CMakeFiles/parquet_shared.dir/Unity/unity_3_cxx.cxx:16:
   D:/a/arrow/arrow/cpp/src/parquet/schema.cc: In function 'void parquet::schema::PrintRepLevel(parquet::Repetition::type, std::ostream&)':
   D:/a/arrow/arrow/cpp/src/parquet/schema.cc:630:30: error: expected unqualified-id before ':' token
     630 |     case Repetition::OPTIONAL:
   ```
   
   https://github.com/apache/arrow/actions/runs/14115826826/job/39545693922?pr=45360#step:7:838
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2026590866


##########
cpp/src/parquet/properties.h:
##########
@@ -245,6 +245,14 @@ class PARQUET_EXPORT ColumnProperties {
   bool page_index_enabled_;
 };
 
+struct CDCOptions {
+  int64_t min_chunk_size;
+  int64_t max_chunk_size;
+  int8_t norm_factor;
+};
+
+static constexpr CDCOptions kDefaultCdcOptions = CDCOptions{256 * 1024, 1024 * 1024, 0};

Review Comment:
   > Regarding the data page size, the default is
   > 
   > ```c++
   > static constexpr int64_t kDefaultDataPageSize = 1024 * 1024;
   > ```
   
   Hmm, I'm getting confused about the [desired data page size](https://lists.apache.org/thread/vsxmbvnx9gy5414cfo25mnwcj17h1xyp) again. @wgtmac Can you enlighten me/us here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027837184


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();
+    auto min_length = ElementCount(min_chunk_size, byte_width, nullable);
+    auto max_length = ElementCount(max_chunk_size, byte_width, nullable);
+    AssertAllBetween(base_info.page_lengths, min_length, max_length);
+    AssertAllBetween(modified_info.page_lengths, min_length, max_length);
+  } else if (::arrow::is_base_binary_like(dtype->id()) && !enable_dictionary) {
+    AssertAllBetween(base_info.page_sizes, min_chunk_size, max_chunk_size);
+    AssertAllBetween(modified_info.page_sizes, min_chunk_size, max_chunk_size);
+  }
+}
+
+constexpr int64_t kMinChunkSize = 8 * 1024;
+constexpr int64_t kMaxChunkSize = 32 * 1024;
+constexpr int64_t kPartSize = 128 * 1024;
+constexpr int64_t kEditSize = 128;
+
+struct CaseConfig {
+  // Arrow data type to generate the testing data for
+  std::shared_ptr<::arrow::DataType> dtype;
+  // Whether the data type is nullable
+  bool is_nullable;
+  // Approximate number of bytes per record to calculate the number of elements to
+  // generate
+  size_t bytes_per_record;
+  // Data page version to use
+  ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1;
+};
+
+// Define PrintTo for MyStruct
+void PrintTo(const CaseConfig& param, std::ostream* os) {
+  *os << "{ " << param.dtype->ToString();
+  if (param.is_nullable) {
+    *os << " nullable";
+  }
+  *os << " }";
+}
+
+class TestCDC : public ::testing::Test {
+ public:
+  uint64_t GetMask(const ContentDefinedChunker& cdc) const { return cdc.GetMask(); }
+};
+
+TEST_F(TestCDC, RollingHashMaskCalculation) {
+  auto le = LevelInfo();
+  auto min_size = 256 * 1024;
+  auto max_size = 1024 * 1024;
+
+  auto cdc0 = ContentDefinedChunker(le, min_size, max_size, 0);
+  ASSERT_EQ(GetMask(cdc0), 0xFFFE000000000000);
+
+  auto cdc1 = ContentDefinedChunker(le, min_size, max_size, 1);
+  ASSERT_EQ(GetMask(cdc1), 0xFFFC000000000000);
+
+  auto cdc2 = ContentDefinedChunker(le, min_size, max_size, 2);
+  ASSERT_EQ(GetMask(cdc2), 0xFFF8000000000000);
+
+  auto cdc3 = ContentDefinedChunker(le, min_size, max_size, 3);
+  ASSERT_EQ(GetMask(cdc3), 0xFFF0000000000000);
+
+  auto cdc4 = ContentDefinedChunker(le, min_size, max_size, -1);
+  ASSERT_EQ(GetMask(cdc4), 0xFFFF000000000000);
+
+  // this is the smallest possible mask always matching, by using 8 hashtables
+  // we are going to have a match every 8 bytes; this is an unrealistic case
+  // but checking for the correctness of the mask calculation
+  auto cdc5 = ContentDefinedChunker(le, 0, 16, 0);
+  ASSERT_EQ(GetMask(cdc5), 0x0000000000000000);
+
+  auto cdc6 = ContentDefinedChunker(le, 0, 32, 1);
+  ASSERT_EQ(GetMask(cdc6), 0x0000000000000000);
+
+  auto cdc7 = ContentDefinedChunker(le, 0, 16, -1);
+  ASSERT_EQ(GetMask(cdc7), 0x8000000000000000);
+
+  // another unrealistic case, checking for the validation
+  auto cdc8 = ContentDefinedChunker(le, 128, 384, -60);
+  ASSERT_EQ(GetMask(cdc8), 0xFFFFFFFFFFFFFFFF);
+}
+
+TEST_F(TestCDC, WriteSingleColumnParquetFile) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  std::vector<int32_t> numbers = {1, 2, 3, 4, 5};
+  std::vector<uint8_t> valid_bits = {1, 0, 1, 0, 1};
+  EXPECT_THROW(
+      int_column_writer.WriteBatch(numbers.size(), nullptr, nullptr, numbers.data()),
+      ParquetException);
+  EXPECT_THROW(int_column_writer.WriteBatchSpaced(numbers.size(), nullptr, nullptr,
+                                                  valid_bits.data(), 0, numbers.data()),
+               ParquetException);
+}
+
+TEST_F(TestCDC, LastChunkDoesntTriggerAddDataPage) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()
+                   ->content_defined_chunking_options(kMinChunkSize, kMaxChunkSize, 0)
+                   ->disable_dictionary()
+                   ->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  ASSERT_OK_AND_ASSIGN(auto array, ::arrow::gen::Step()->Generate(8000));
+  auto arrow_props = default_arrow_writer_properties();
+  auto arrow_ctx = ArrowWriteContext(default_memory_pool(), arrow_props.get());
+
+  // Calling WriteArrow twice, we expect that the first call doesn't add a new data page
+  // at the end allowing subsequent calls to append to the same page
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+
+  int_column_writer.Close();
+  writer->Close();
+  ASSERT_OK_AND_ASSIGN(auto buffer, sink->Finish());
+
+  auto info = GetColumnParquetInfo(buffer);
+  ASSERT_EQ(info.size(), 1);
+
+  // AssertAllBetween allow the last chunk size to be smaller than the min_chunk_size
+  AssertAllBetween(info[0].page_sizes, kMinChunkSize, kMaxChunkSize);
+  AssertAllBetween(info[0].page_lengths, 3000, 5000);
+}
+
+TEST_F(TestCDC, ChunkSizeParameterValidation) {
+  // Test that constructor validates min/max chunk size parameters
+  auto li = LevelInfo();
+
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 256 * 1024, 1024 * 1024));
+
+  // with norm_factor=0 the difference between min and max chunk size must be
+  // at least 16
+  ASSERT_THROW(ContentDefinedChunker(li, 0, -1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1024, 512), ParquetException);
+
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16));
+  ASSERT_THROW(ContentDefinedChunker(li, -16, -16), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 16, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 32, 32), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 32, 48));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1024 * 1024, 2 * 1024 * 1024));
+  ASSERT_NO_THROW(
+      ContentDefinedChunker(li, 1024 * 1024 * 1024L, 2LL * 1024 * 1024 * 1024L));
+
+  // with norm_factor=1 the difference between min and max chunk size must be
+  // at least 64
+  ASSERT_THROW(ContentDefinedChunker(li, 1, -1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 32, 1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1, 33, 1));
+
+  // with norm_factor=2 the difference between min and max chunk size must be
+  // at least 128
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 63, 2), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 64, 2));
+
+  // with norm_factor=-1 the difference between min and max chunk size must be
+  // at least 8
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 7, -1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 8, -1));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16, -1));
+
+  // test the norm_factor extremes
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0, -68), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 1, -67));
+  ASSERT_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 59),
+               ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 58));
+}
+
+class TestCDCSingleRowGroup : public ::testing::TestWithParam<CaseConfig> {
+ protected:
+  // Column random table parts for testing
+  std::shared_ptr<Field> field_;
+  std::shared_ptr<Table> part1_, part2_, part3_, part4_, part5_, part6_, part7_;
+
+  void SetUp() override {
+    const auto& param = GetParam();
+    auto field_ = ::arrow::field("f0", param.dtype, param.is_nullable);
+    auto schema = ::arrow::schema({field_});
+
+    auto part_length = kPartSize / param.bytes_per_record;
+    auto edit_length = kEditSize / param.bytes_per_record;
+    ASSERT_OK_AND_ASSIGN(part1_, GenerateTable(schema, part_length, 0));
+    ASSERT_OK_AND_ASSIGN(part2_, GenerateTable(schema, edit_length, 1));
+    ASSERT_OK_AND_ASSIGN(part3_, GenerateTable(schema, part_length, part_length));
+    ASSERT_OK_AND_ASSIGN(part4_, GenerateTable(schema, edit_length, 2));
+    ASSERT_OK_AND_ASSIGN(part5_, GenerateTable(schema, part_length, 2 * part_length));
+    ASSERT_OK_AND_ASSIGN(part6_, GenerateTable(schema, edit_length, 3));
+    ASSERT_OK_AND_ASSIGN(part7_, GenerateTable(schema, edit_length, 4));
+  }
+};
+
+TEST_P(TestCDCSingleRowGroup, DeleteOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/0,
+                                /*exact_number_of_smaller_diffs=*/1, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, DeleteTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part3_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/0,
+                                /*exact_number_of_smaller_diffs=*/2, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, UpdateOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part4_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*max_number_of_equal_diffs=*/1);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, UpdateTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified,
+                       ConcatAndCombine({part1_, part6_, part3_, part7_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*max_number_of_equal_diffs=*/2);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, InsertOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/1,
+                                /*exact_number_of_smaller_diffs=*/0, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, InsertTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part3_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/2,
+                                /*exact_number_of_smaller_diffs=*/0, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, Append) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part2_, part3_, part4_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto original_page_lengths = base_info.front().page_lengths;
+    auto modified_page_lengths = modified_info.front().page_lengths;
+    ASSERT_GE(original_page_lengths.size(), modified_page_lengths.size());

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027850653


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because

Review Comment:
   Not relevant anymore, removed this code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027843116


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);

Review Comment:
   Yes, this is an assumption by the assertion function, I added a comment about this with a note to improve it in the future. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2083263612


##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1249,6 +1258,16 @@ class TypedColumnWriterImpl : public ColumnWriterImpl,
 
   int64_t WriteBatch(int64_t num_values, const int16_t* def_levels,
                      const int16_t* rep_levels, const T* values) override {
+    if (properties_->content_defined_chunking_enabled()) {

Review Comment:
   Sure, added `ARROW_PREDICT_FALSE` to these branches.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["wgtmac (via GitHub)" <gi...@apache.org>]

wgtmac commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2085763204


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,412 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+namespace {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_level parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_level Normalization level (default 0)
+// @return The mask used to compare against the rolling hash
+uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size, int norm_level) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be non-negative");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes; also divide by the number of gearhash tables to have a
+  // a more gaussian-like distribution
+  int64_t target_size = (avg_chunk_size - min_chunk_size) / kNumGearhashTables;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // a user defined `norm_level` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_level we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_level is 0 by
+  // default
+  int effective_bits = mask_bits - norm_level;
+
+  if (effective_bits < 1 || effective_bits > 63) {
+    throw ParquetException(
+        "The number of bits in the CDC mask must be between 1 and 63, got " +
+        std::to_string(effective_bits));
+  } else {
+    // create the mask by setting the top bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+}  // namespace
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int norm_level)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_level)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the

Review Comment:
   I don't have a strong preference and slightly feel inclined to a).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1963655008


##########
cpp/src/parquet/column_chunker.h:
##########
@@ -0,0 +1,784 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+using arrow::internal::checked_cast;
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+const int64_t AVG_LEN = 1024 * 1024;
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t avg_len, size_t adjustement_level) {
+  size_t mask_bits = static_cast<size_t>(std::ceil(std::log2(avg_len)));
+  return (1ULL << (mask_bits - adjustement_level)) - 1;
+}
+
+// rename it since it is not FastCDC anymore
+class FastCDC {
+ public:
+  FastCDC(const LevelInfo& level_info, uint64_t avg_len, uint8_t granurality_level = 5)
+      : level_info_(level_info),
+        avg_len_(avg_len == 0 ? AVG_LEN : avg_len),
+        min_len_(static_cast<uint64_t>(avg_len_ * 0.6)),

Review Comment:
   Exposed min and max only.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1974000251


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1981155077


##########
cpp/src/parquet/column_chunker.h:
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+
+namespace internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage systems (CAS)
+/// which split the bytes stream into content defined blobs. The CAS system will calculate
+/// a unique identifier for each blob, then store the blob in a key-value store. If the
+/// same blob is encountered again, the system can refer to the hash instead of physically
+/// storing the blob again. In the example above, the CAS system would phiysically store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks///

Review Comment:
   Done.



##########
cpp/src/parquet/column_writer.h:
##########
@@ -23,6 +23,7 @@
 
 #include "arrow/type_fwd.h"
 #include "arrow/util/compression.h"
+#include "parquet/column_chunker.h"

Review Comment:
   Removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1983354362


##########
cpp/src/parquet/chunker_internal_hashtable.h:
##########
@@ -0,0 +1,545 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+#include <cstdint>
+
+namespace parquet::internal {
+
+constexpr uint64_t GEARHASH_TABLE[8][256] = {

Review Comment:
   Let's please use the modern naming convention `kGearhashTable`?



##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage system (CAS)
+/// which splits the bytes stream into content defined blobs. The CAS system will
+/// calculate a unique identifier for each blob, then store the blob in a key-value store.
+/// If the same blob is encountered again, the system can refer to the hash instead of
+/// physically storing the blob again. In the example above, the CAS system would store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks.
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data
+///   Deduplication
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+/// - Git is for Data (chunk size normalization used here is described in section 6.2.1):
+///   https://www.cidrdb.org/cidr2023/papers/p43-low.pdf
+class ContentDefinedChunker {
+ public:
+  /// Create a new ContentDefinedChunker instance
+  ///
+  /// @param level_info Information about definition and repetition levels
+  /// @param size_range Min/max chunk size as pair<min_size, max_size>, the chunker will
+  ///                   attempt to uniformly distribute the chunks between these extremes.
+  /// @param norm_factor Normalization factor to center the chunk size around the average
+  ///                    size more aggressively. By increasing the normalization factor,
+  ///                    probability of finding a chunk boundary increases.

Review Comment:
   So is there a drawback in increasing the normalization factor? Why can't I just pass 255 always?
   
   (also, why is this `uint8_t`?)



##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {

Review Comment:
   If the only motivation is to generate random data, then let's please use `RandomArrayGenerator` with a fixed seed instead? 



##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, uint64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal128Builder, ::arrow::Decimal128(val % max_value))
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal256Builder, ::arrow::Decimal256(val % max_value))
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      GENERATE_CASE_BODY(::arrow::FixedSizeBinaryBuilder,
+                         std::string("bin_") + std::to_string(val).substr(0, size - 4))
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, uint64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(const std::shared_ptr<Table>& table,
+                                                   uint64_t min_chunk_size,
+                                                   uint64_t max_chunk_size,
+                                                   bool enable_dictionary = false,
+
+                                                   int64_t row_group_size = 1024 * 1024) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_cdc()
+      ->cdc_size_range(min_chunk_size, max_chunk_size)
+      ->cdc_norm_factor(0);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+struct PageSizes {
+  std::vector<uint64_t> lengths;
+  std::vector<uint64_t> sizes;
+};
+
+PageSizes GetColumnPageSizes(const std::shared_ptr<Buffer>& data, int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  PageSizes result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        result.sizes.push_back(data_page->size());
+        result.lengths.push_back(data_page->num_values());
+      }
+    }
+  }
+
+  return result;
+}
+
+Result<PageSizes> WriteAndGetPageSizes(const std::shared_ptr<Table>& table,
+                                       uint64_t min_chunk_size, uint64_t max_chunk_size,
+                                       bool enable_dictionary = false,
+                                       int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnPageSizes(buffer, column_index);
+}
+
+void AssertAllBetween(const std::vector<uint64_t>& values, uint64_t min, uint64_t max,
+                      bool expect_dictionary_fallback = false) {
+  // expect the last chunk since it is not guaranteed to be within the range
+  if (expect_dictionary_fallback) {
+    // if dictionary encoding is enabled, the writer can fallback to plain
+    // encoding splitting within a content defined chunk, so we can't
+    // guarantee that all chunks are within the range in this case, but we
+    // know that there can be at most 2 pages smaller than the min_chunk_size
+    size_t smaller_count = 0;
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      if (values[i] < min) {
+        smaller_count++;
+      } else {
+        ASSERT_LE(values[i], max);
+      }
+    }
+    ASSERT_LE(smaller_count, 2);
+  } else {
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      ASSERT_GE(values[i], min);
+      ASSERT_LE(values[i], max);
+    }
+  }
+  ASSERT_LE(values.back(), max);
+}
+
+std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>> FindDifferences(
+    const std::vector<uint64_t>& first, const std::vector<uint64_t>& second) {
+  // Compute LCS table.
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack to get common indices.
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      i--;
+    } else {
+      j--;
+    }
+  }
+  std::reverse(common.begin(), common.end());
+
+  // Build raw differences.
+  std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Merge adjacent diffs if one side is empty in the first diff and the other side
+  // is empty in the next diff, to avoid splitting single changes into two parts.
+  std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+      if (can_merge_a) {
+        // Combine into one change
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    merged.push_back(std::move(diff));
+  }
+
+  return merged;
+}
+
+void PrintDifferences(
+    const std::vector<uint64_t>& original, const std::vector<uint64_t>& modified,
+    std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>>& diffs) {
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  std::vector<uint64_t> first = {1, 2, 3, 4, 5};
+  std::vector<uint64_t> second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({2, 3}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  std::vector<uint64_t> first = {1, 2, 3, 4, 5, 6, 7};
+  std::vector<uint64_t> second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({2, 3}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, std::vector<uint64_t>({5}));
+  ASSERT_EQ(diffs[1].second, std::vector<uint64_t>({10}));
+
+  ASSERT_EQ(diffs[2].first, std::vector<uint64_t>({7}));
+  ASSERT_EQ(diffs[2].second, std::vector<uint64_t>({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  std::vector<uint64_t> first = {1, 2, 3};
+  std::vector<uint64_t> second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  std::vector<uint64_t> first = {};
+  std::vector<uint64_t> second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  std::vector<uint64_t> first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  std::vector<uint64_t> second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  std::vector<uint64_t> first = {2169, 1976, 2180, 2147, 1934, 1772,
+                                 1914, 2075, 2154, 1940, 1934, 1970};
+  std::vector<uint64_t> second = {2169, 1976, 2180, 2147, 2265, 1804,
+                                  1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  std::vector<uint64_t> original = {445, 312, 393, 401, 410, 138, 558, 457};
+  std::vector<uint64_t> modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({401}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertUpdateCase(const std::shared_ptr<::arrow::DataType>& dtype,

Review Comment:
   Can you add a comment describing this function?



##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}

Review Comment:
   This is not needed AFAICT.



##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage system (CAS)
+/// which splits the bytes stream into content defined blobs. The CAS system will
+/// calculate a unique identifier for each blob, then store the blob in a key-value store.
+/// If the same blob is encountered again, the system can refer to the hash instead of
+/// physically storing the blob again. In the example above, the CAS system would store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks.
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data
+///   Deduplication
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+/// - Git is for Data (chunk size normalization used here is described in section 6.2.1):
+///   https://www.cidrdb.org/cidr2023/papers/p43-low.pdf
+class ContentDefinedChunker {
+ public:
+  /// Create a new ContentDefinedChunker instance
+  ///
+  /// @param level_info Information about definition and repetition levels
+  /// @param size_range Min/max chunk size as pair<min_size, max_size>, the chunker will
+  ///                   attempt to uniformly distribute the chunks between these extremes.
+  /// @param norm_factor Normalization factor to center the chunk size around the average
+  ///                    size more aggressively. By increasing the normalization factor,
+  ///                    probability of finding a chunk boundary increases.
+  ContentDefinedChunker(const LevelInfo& level_info,
+                        std::pair<uint64_t, uint64_t> size_range,
+                        uint8_t norm_factor = 0);
+
+  /// Get the chunk boundaries for the given column data
+  ///
+  /// @param def_levels Definition levels
+  /// @param rep_levels Repetition levels
+  /// @param num_levels Number of levels
+  /// @param values Column values as an Arrow array
+  /// @return Vector of Chunk objects representing the chunk boundaries
+  const std::vector<Chunk> GetBoundaries(const int16_t* def_levels,
+                                         const int16_t* rep_levels, int64_t num_levels,
+                                         const ::arrow::Array& values);
+
+ private:
+  // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+  // true if the hash matches the mask.
+  template <typename T>
+  void Roll(const T value);
+

Review Comment:
   Ok, instead of declaring private methods in the `.h`, we should use the pimpl idiom and hide the implementation entirely, e.g.:
   ```suggestion
    private:
     class Impl;
     std::unique_ptr<Impl> impl_;
   };
   ```



##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage system (CAS)
+/// which splits the bytes stream into content defined blobs. The CAS system will
+/// calculate a unique identifier for each blob, then store the blob in a key-value store.
+/// If the same blob is encountered again, the system can refer to the hash instead of
+/// physically storing the blob again. In the example above, the CAS system would store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks.
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data
+///   Deduplication
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+/// - Git is for Data (chunk size normalization used here is described in section 6.2.1):
+///   https://www.cidrdb.org/cidr2023/papers/p43-low.pdf
+class ContentDefinedChunker {
+ public:
+  /// Create a new ContentDefinedChunker instance
+  ///
+  /// @param level_info Information about definition and repetition levels
+  /// @param size_range Min/max chunk size as pair<min_size, max_size>, the chunker will
+  ///                   attempt to uniformly distribute the chunks between these extremes.

Review Comment:
   Is it in bytes or values? Let's be explicit in the docstring.



##########
cpp/src/parquet/chunker_internal_hashtable.py:
##########
@@ -0,0 +1,90 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import hashlib
+import pathlib
+import sys
+from io import StringIO
+
+
+template = """\
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+#include <cstdint>
+
+namespace parquet {{
+namespace internal {{
+
+constexpr uint64_t GEARHASH_TABLE[8][256] = {{
+{content}}};
+
+}}  // namespace internal
+}}  // namespace parquet
+"""
+
+
+def generate_hash(n: int, seed: int):

Review Comment:
   Can you add a comment explaining/giving a reference to the algorithm used for computing this?



##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, uint64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal128Builder, ::arrow::Decimal128(val % max_value))
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal256Builder, ::arrow::Decimal256(val % max_value))
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      GENERATE_CASE_BODY(::arrow::FixedSizeBinaryBuilder,
+                         std::string("bin_") + std::to_string(val).substr(0, size - 4))
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, uint64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(const std::shared_ptr<Table>& table,
+                                                   uint64_t min_chunk_size,
+                                                   uint64_t max_chunk_size,
+                                                   bool enable_dictionary = false,
+
+                                                   int64_t row_group_size = 1024 * 1024) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_cdc()
+      ->cdc_size_range(min_chunk_size, max_chunk_size)
+      ->cdc_norm_factor(0);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+struct PageSizes {
+  std::vector<uint64_t> lengths;
+  std::vector<uint64_t> sizes;
+};
+
+PageSizes GetColumnPageSizes(const std::shared_ptr<Buffer>& data, int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  PageSizes result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        result.sizes.push_back(data_page->size());
+        result.lengths.push_back(data_page->num_values());
+      }
+    }
+  }
+
+  return result;
+}
+
+Result<PageSizes> WriteAndGetPageSizes(const std::shared_ptr<Table>& table,
+                                       uint64_t min_chunk_size, uint64_t max_chunk_size,
+                                       bool enable_dictionary = false,
+                                       int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnPageSizes(buffer, column_index);
+}
+
+void AssertAllBetween(const std::vector<uint64_t>& values, uint64_t min, uint64_t max,
+                      bool expect_dictionary_fallback = false) {
+  // expect the last chunk since it is not guaranteed to be within the range
+  if (expect_dictionary_fallback) {
+    // if dictionary encoding is enabled, the writer can fallback to plain
+    // encoding splitting within a content defined chunk, so we can't
+    // guarantee that all chunks are within the range in this case, but we
+    // know that there can be at most 2 pages smaller than the min_chunk_size
+    size_t smaller_count = 0;
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      if (values[i] < min) {
+        smaller_count++;
+      } else {
+        ASSERT_LE(values[i], max);
+      }
+    }
+    ASSERT_LE(smaller_count, 2);
+  } else {
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      ASSERT_GE(values[i], min);
+      ASSERT_LE(values[i], max);
+    }
+  }
+  ASSERT_LE(values.back(), max);
+}
+
+std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>> FindDifferences(
+    const std::vector<uint64_t>& first, const std::vector<uint64_t>& second) {
+  // Compute LCS table.

Review Comment:
   LCS?



##########
cpp/src/parquet/chunker_internal_hashtable.h:
##########
@@ -0,0 +1,545 @@
+// Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   If this file is generated, can you:
   1) name it `chunked_internal_generated.h`
   2) _and_ mark it generated [in the git attributes](https://docs.github.com/en/repositories/working-with-files/managing-files/customizing-how-changed-files-appear-on-github)
   
   ?



##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage system (CAS)
+/// which splits the bytes stream into content defined blobs. The CAS system will
+/// calculate a unique identifier for each blob, then store the blob in a key-value store.
+/// If the same blob is encountered again, the system can refer to the hash instead of
+/// physically storing the blob again. In the example above, the CAS system would store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks.
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data
+///   Deduplication
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+/// - Git is for Data (chunk size normalization used here is described in section 6.2.1):
+///   https://www.cidrdb.org/cidr2023/papers/p43-low.pdf
+class ContentDefinedChunker {
+ public:
+  /// Create a new ContentDefinedChunker instance
+  ///
+  /// @param level_info Information about definition and repetition levels
+  /// @param size_range Min/max chunk size as pair<min_size, max_size>, the chunker will
+  ///                   attempt to uniformly distribute the chunks between these extremes.
+  /// @param norm_factor Normalization factor to center the chunk size around the average
+  ///                    size more aggressively. By increasing the normalization factor,
+  ///                    probability of finding a chunk boundary increases.
+  ContentDefinedChunker(const LevelInfo& level_info,
+                        std::pair<uint64_t, uint64_t> size_range,
+                        uint8_t norm_factor = 0);

Review Comment:
   Nit, but I'd rather have:
   ```suggestion
     ContentDefinedChunker(const LevelInfo& level_info,
                           int64_t min_size, int64_t max_size,
                           uint8_t norm_factor = 0);
   ```
   



##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, uint64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal128Builder, ::arrow::Decimal128(val % max_value))
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal256Builder, ::arrow::Decimal256(val % max_value))
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      GENERATE_CASE_BODY(::arrow::FixedSizeBinaryBuilder,
+                         std::string("bin_") + std::to_string(val).substr(0, size - 4))
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, uint64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(const std::shared_ptr<Table>& table,
+                                                   uint64_t min_chunk_size,
+                                                   uint64_t max_chunk_size,
+                                                   bool enable_dictionary = false,
+
+                                                   int64_t row_group_size = 1024 * 1024) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_cdc()
+      ->cdc_size_range(min_chunk_size, max_chunk_size)
+      ->cdc_norm_factor(0);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+struct PageSizes {
+  std::vector<uint64_t> lengths;
+  std::vector<uint64_t> sizes;
+};
+
+PageSizes GetColumnPageSizes(const std::shared_ptr<Buffer>& data, int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  PageSizes result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        result.sizes.push_back(data_page->size());
+        result.lengths.push_back(data_page->num_values());
+      }
+    }
+  }
+
+  return result;
+}
+
+Result<PageSizes> WriteAndGetPageSizes(const std::shared_ptr<Table>& table,
+                                       uint64_t min_chunk_size, uint64_t max_chunk_size,
+                                       bool enable_dictionary = false,
+                                       int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnPageSizes(buffer, column_index);
+}
+
+void AssertAllBetween(const std::vector<uint64_t>& values, uint64_t min, uint64_t max,
+                      bool expect_dictionary_fallback = false) {
+  // expect the last chunk since it is not guaranteed to be within the range
+  if (expect_dictionary_fallback) {
+    // if dictionary encoding is enabled, the writer can fallback to plain
+    // encoding splitting within a content defined chunk, so we can't
+    // guarantee that all chunks are within the range in this case, but we
+    // know that there can be at most 2 pages smaller than the min_chunk_size
+    size_t smaller_count = 0;
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      if (values[i] < min) {
+        smaller_count++;
+      } else {
+        ASSERT_LE(values[i], max);
+      }
+    }
+    ASSERT_LE(smaller_count, 2);
+  } else {
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      ASSERT_GE(values[i], min);
+      ASSERT_LE(values[i], max);
+    }
+  }
+  ASSERT_LE(values.back(), max);
+}
+
+std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>> FindDifferences(
+    const std::vector<uint64_t>& first, const std::vector<uint64_t>& second) {
+  // Compute LCS table.
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack to get common indices.
+  std::vector<std::pair<size_t, size_t>> common;

Review Comment:
   Same here: if the pair's contents are not obvious, let's define a struct?



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,250 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_hashtable.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {

Review Comment:
   Can you add a comment referencing the algorithm used here?



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,250 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_hashtable.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size

Review Comment:
   Is it actually "gaussian-like"?



##########
cpp/src/parquet/chunker_internal_hashtable.py:
##########
@@ -0,0 +1,90 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import hashlib
+import pathlib
+import sys
+from io import StringIO
+
+
+template = """\
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+#include <cstdint>
+
+namespace parquet {{
+namespace internal {{
+
+constexpr uint64_t GEARHASH_TABLE[8][256] = {{
+{content}}};
+
+}}  // namespace internal
+}}  // namespace parquet
+"""
+
+
+def generate_hash(n: int, seed: int):
+    value = bytes([seed] * 64 + [n] * 64)
+    hasher = hashlib.md5(value)
+    return hasher.hexdigest()[:16]
+
+
+def generate_hashtable(seed: int, length=256, comma=True):

Review Comment:
   `comma` isn't used below?



##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, uint64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal128Builder, ::arrow::Decimal128(val % max_value))
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal256Builder, ::arrow::Decimal256(val % max_value))
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      GENERATE_CASE_BODY(::arrow::FixedSizeBinaryBuilder,
+                         std::string("bin_") + std::to_string(val).substr(0, size - 4))
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, uint64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(const std::shared_ptr<Table>& table,
+                                                   uint64_t min_chunk_size,
+                                                   uint64_t max_chunk_size,
+                                                   bool enable_dictionary = false,
+
+                                                   int64_t row_group_size = 1024 * 1024) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_cdc()
+      ->cdc_size_range(min_chunk_size, max_chunk_size)
+      ->cdc_norm_factor(0);

Review Comment:
   It might be better to define a `struct CDCOptions` so that we don't have too many CDC-related methods on `WriterProperties`.



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,250 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_hashtable.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][bytes[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);

Review Comment:
   I'm curious, why match at each byte? Shouldn't you match at the end of the value instead?



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,250 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_hashtable.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][bytes[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);

Review Comment:
   Same question here.



##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \

Review Comment:
   Wow, please define a templated function instead of this.



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,250 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_hashtable.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][bytes[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling
+    // hash state here, so the next NeedNewChunk() call will continue from the current
+    // state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename T>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const T& leaf_array) {
+  std::vector<Chunk> chunks;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      chunks.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(def_levels[offset]);
+      Roll(leaf_array.GetView(offset));

Review Comment:
   So, instead, you might want to do something like this:
   ```suggestion
         if (def_levels[offset] == max_def_level) {
           Roll(leaf_array.GetView(offset));
         }
   ```



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,250 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_hashtable.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][bytes[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling
+    // hash state here, so the next NeedNewChunk() call will continue from the current
+    // state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename T>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const T& leaf_array) {
+  std::vector<Chunk> chunks;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      chunks.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(def_levels[offset]);
+      Roll(leaf_array.GetView(offset));

Review Comment:
   Er, you shouldn't call `GetView` if the array element is null, because the raw value is undefined (and won't be written out to Parquet anyway).



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,250 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_hashtable.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][bytes[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling

Review Comment:
   ```suggestion
       // we have a hard limit on the maximum chunk size, note that we don't reset the rolling
   ```



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,250 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_hashtable.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][bytes[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling
+    // hash state here, so the next NeedNewChunk() call will continue from the current
+    // state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename T>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const T& leaf_array) {
+  std::vector<Chunk> chunks;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      chunks.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(def_levels[offset]);
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      chunks.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else {
+    // nested data with nulls
+    bool has_leaf_value;
+    bool is_record_boundary;
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t value_offset = 0;
+    int64_t record_level_offset = 0;
+    int64_t record_value_offset = 0;
+
+    for (int64_t level_offset = 0; level_offset < num_levels; ++level_offset) {
+      def_level = def_levels[level_offset];
+      rep_level = rep_levels[level_offset];
+
+      has_leaf_value = def_level >= level_info_.repeated_ancestor_def_level;
+      is_record_boundary = rep_level == 0;
+
+      Roll(def_level);
+      Roll(rep_level);
+      if (has_leaf_value) {
+        Roll(leaf_array.GetView(value_offset));

Review Comment:
   Same as above.



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,250 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_hashtable.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][bytes[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling
+    // hash state here, so the next NeedNewChunk() call will continue from the current
+    // state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename T>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const T& leaf_array) {
+  std::vector<Chunk> chunks;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      chunks.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(def_levels[offset]);
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      chunks.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else {
+    // nested data with nulls
+    bool has_leaf_value;
+    bool is_record_boundary;
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t value_offset = 0;
+    int64_t record_level_offset = 0;
+    int64_t record_value_offset = 0;
+
+    for (int64_t level_offset = 0; level_offset < num_levels; ++level_offset) {
+      def_level = def_levels[level_offset];
+      rep_level = rep_levels[level_offset];
+
+      has_leaf_value = def_level >= level_info_.repeated_ancestor_def_level;
+      is_record_boundary = rep_level == 0;
+
+      Roll(def_level);
+      Roll(rep_level);
+      if (has_leaf_value) {
+        Roll(leaf_array.GetView(value_offset));
+      }
+
+      if (is_record_boundary && NeedNewChunk()) {
+        auto levels_to_write = level_offset - record_level_offset;
+        if (levels_to_write > 0) {
+          chunks.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+          record_level_offset = level_offset;
+          record_value_offset = value_offset;
+        }
+      }
+
+      if (has_leaf_value) {
+        ++value_offset;
+      }
+    }
+
+    auto levels_to_write = num_levels - record_level_offset;
+    if (levels_to_write > 0) {
+      chunks.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+    }
+  }
+
+  return chunks;
+}
+
+#define PRIMITIVE_CASE(TYPE_ID, ArrowType)               \
+  case ::arrow::Type::TYPE_ID:                           \
+    return Calculate(def_levels, rep_levels, num_levels, \
+                     static_cast<const ::arrow::ArrowType##Array&>(values));
+

Review Comment:
   Generating a separate `Calculate` for each primitive type is going to be wasteful. Some implementations should be shared, for example all 64-bit numeric types (int64, uint64, float64, timestamp, etc.). Same for string/binary, large_string/large_binary...



##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, uint64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal128Builder, ::arrow::Decimal128(val % max_value))
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal256Builder, ::arrow::Decimal256(val % max_value))
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      GENERATE_CASE_BODY(::arrow::FixedSizeBinaryBuilder,
+                         std::string("bin_") + std::to_string(val).substr(0, size - 4))
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, uint64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(const std::shared_ptr<Table>& table,
+                                                   uint64_t min_chunk_size,
+                                                   uint64_t max_chunk_size,

Review Comment:
   Can we use  `int64_t` everywhere for sizes, not `uint64_t`? This is the convention accros the codebase.



##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, uint64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal128Builder, ::arrow::Decimal128(val % max_value))
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal256Builder, ::arrow::Decimal256(val % max_value))
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      GENERATE_CASE_BODY(::arrow::FixedSizeBinaryBuilder,
+                         std::string("bin_") + std::to_string(val).substr(0, size - 4))
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, uint64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(const std::shared_ptr<Table>& table,
+                                                   uint64_t min_chunk_size,
+                                                   uint64_t max_chunk_size,
+                                                   bool enable_dictionary = false,
+
+                                                   int64_t row_group_size = 1024 * 1024) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_cdc()
+      ->cdc_size_range(min_chunk_size, max_chunk_size)
+      ->cdc_norm_factor(0);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+struct PageSizes {
+  std::vector<uint64_t> lengths;
+  std::vector<uint64_t> sizes;
+};
+
+PageSizes GetColumnPageSizes(const std::shared_ptr<Buffer>& data, int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  PageSizes result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        result.sizes.push_back(data_page->size());
+        result.lengths.push_back(data_page->num_values());
+      }
+    }
+  }
+
+  return result;
+}
+
+Result<PageSizes> WriteAndGetPageSizes(const std::shared_ptr<Table>& table,
+                                       uint64_t min_chunk_size, uint64_t max_chunk_size,
+                                       bool enable_dictionary = false,
+                                       int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnPageSizes(buffer, column_index);
+}
+
+void AssertAllBetween(const std::vector<uint64_t>& values, uint64_t min, uint64_t max,
+                      bool expect_dictionary_fallback = false) {
+  // expect the last chunk since it is not guaranteed to be within the range
+  if (expect_dictionary_fallback) {
+    // if dictionary encoding is enabled, the writer can fallback to plain
+    // encoding splitting within a content defined chunk, so we can't
+    // guarantee that all chunks are within the range in this case, but we
+    // know that there can be at most 2 pages smaller than the min_chunk_size
+    size_t smaller_count = 0;
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      if (values[i] < min) {
+        smaller_count++;
+      } else {
+        ASSERT_LE(values[i], max);
+      }
+    }
+    ASSERT_LE(smaller_count, 2);
+  } else {
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      ASSERT_GE(values[i], min);
+      ASSERT_LE(values[i], max);
+    }
+  }
+  ASSERT_LE(values.back(), max);
+}
+
+std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>> FindDifferences(

Review Comment:
   The `std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>>` is gonna be a bit cryptic, can you instead define struct types with meaningful names?



##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, uint64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal128Builder, ::arrow::Decimal128(val % max_value))
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal256Builder, ::arrow::Decimal256(val % max_value))
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      GENERATE_CASE_BODY(::arrow::FixedSizeBinaryBuilder,
+                         std::string("bin_") + std::to_string(val).substr(0, size - 4))
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, uint64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(const std::shared_ptr<Table>& table,
+                                                   uint64_t min_chunk_size,
+                                                   uint64_t max_chunk_size,
+                                                   bool enable_dictionary = false,
+
+                                                   int64_t row_group_size = 1024 * 1024) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_cdc()
+      ->cdc_size_range(min_chunk_size, max_chunk_size)
+      ->cdc_norm_factor(0);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }

Review Comment:
   Note that dictionary encoding will fallback to plain encoding if the dictionary exceeds a certain cardinality. If the data is random-generated, this may actually never exercise chunking dictionary-encoded data. Can you ensure it does? 



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,250 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_hashtable.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][bytes[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling
+    // hash state here, so the next NeedNewChunk() call will continue from the current
+    // state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename T>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const T& leaf_array) {
+  std::vector<Chunk> chunks;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      chunks.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(def_levels[offset]);
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      chunks.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else {
+    // nested data with nulls
+    bool has_leaf_value;
+    bool is_record_boundary;
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t value_offset = 0;
+    int64_t record_level_offset = 0;
+    int64_t record_value_offset = 0;
+
+    for (int64_t level_offset = 0; level_offset < num_levels; ++level_offset) {
+      def_level = def_levels[level_offset];
+      rep_level = rep_levels[level_offset];
+
+      has_leaf_value = def_level >= level_info_.repeated_ancestor_def_level;
+      is_record_boundary = rep_level == 0;
+
+      Roll(def_level);
+      Roll(rep_level);
+      if (has_leaf_value) {
+        Roll(leaf_array.GetView(value_offset));
+      }
+
+      if (is_record_boundary && NeedNewChunk()) {
+        auto levels_to_write = level_offset - record_level_offset;
+        if (levels_to_write > 0) {
+          chunks.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+          record_level_offset = level_offset;
+          record_value_offset = value_offset;
+        }
+      }
+
+      if (has_leaf_value) {
+        ++value_offset;
+      }
+    }
+
+    auto levels_to_write = num_levels - record_level_offset;
+    if (levels_to_write > 0) {
+      chunks.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+    }
+  }
+
+  return chunks;
+}
+
+#define PRIMITIVE_CASE(TYPE_ID, ArrowType)               \
+  case ::arrow::Type::TYPE_ID:                           \
+    return Calculate(def_levels, rep_levels, num_levels, \
+                     static_cast<const ::arrow::ArrowType##Array&>(values));
+
+const std::vector<Chunk> ContentDefinedChunker::GetBoundaries(
+    const int16_t* def_levels, const int16_t* rep_levels, int64_t num_levels,
+    const ::arrow::Array& values) {
+  auto type_id = values.type()->id();
+  switch (type_id) {
+    PRIMITIVE_CASE(BOOL, Boolean)
+    PRIMITIVE_CASE(INT8, Int8)
+    PRIMITIVE_CASE(INT16, Int16)

Review Comment:
   Can we please use something like [`VisitType`](https://github.com/apache/arrow/blob/784aa6faf69f5cf135e09976a281dea9ebf58166/cpp/src/arrow/visit_type_inline.h#L80-L82) instead?



##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, uint64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal128Builder, ::arrow::Decimal128(val % max_value))
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal256Builder, ::arrow::Decimal256(val % max_value))
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      GENERATE_CASE_BODY(::arrow::FixedSizeBinaryBuilder,
+                         std::string("bin_") + std::to_string(val).substr(0, size - 4))
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, uint64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(const std::shared_ptr<Table>& table,
+                                                   uint64_t min_chunk_size,
+                                                   uint64_t max_chunk_size,
+                                                   bool enable_dictionary = false,
+
+                                                   int64_t row_group_size = 1024 * 1024) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_cdc()
+      ->cdc_size_range(min_chunk_size, max_chunk_size)
+      ->cdc_norm_factor(0);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+struct PageSizes {
+  std::vector<uint64_t> lengths;
+  std::vector<uint64_t> sizes;
+};
+
+PageSizes GetColumnPageSizes(const std::shared_ptr<Buffer>& data, int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  PageSizes result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        result.sizes.push_back(data_page->size());
+        result.lengths.push_back(data_page->num_values());
+      }
+    }
+  }
+
+  return result;
+}
+
+Result<PageSizes> WriteAndGetPageSizes(const std::shared_ptr<Table>& table,
+                                       uint64_t min_chunk_size, uint64_t max_chunk_size,
+                                       bool enable_dictionary = false,
+                                       int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnPageSizes(buffer, column_index);
+}
+
+void AssertAllBetween(const std::vector<uint64_t>& values, uint64_t min, uint64_t max,
+                      bool expect_dictionary_fallback = false) {
+  // expect the last chunk since it is not guaranteed to be within the range
+  if (expect_dictionary_fallback) {
+    // if dictionary encoding is enabled, the writer can fallback to plain
+    // encoding splitting within a content defined chunk, so we can't
+    // guarantee that all chunks are within the range in this case, but we
+    // know that there can be at most 2 pages smaller than the min_chunk_size
+    size_t smaller_count = 0;
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      if (values[i] < min) {
+        smaller_count++;
+      } else {
+        ASSERT_LE(values[i], max);
+      }
+    }
+    ASSERT_LE(smaller_count, 2);
+  } else {
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      ASSERT_GE(values[i], min);
+      ASSERT_LE(values[i], max);
+    }
+  }
+  ASSERT_LE(values.back(), max);
+}
+
+std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>> FindDifferences(
+    const std::vector<uint64_t>& first, const std::vector<uint64_t>& second) {
+  // Compute LCS table.
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack to get common indices.
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      i--;
+    } else {
+      j--;
+    }
+  }
+  std::reverse(common.begin(), common.end());
+
+  // Build raw differences.
+  std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Merge adjacent diffs if one side is empty in the first diff and the other side
+  // is empty in the next diff, to avoid splitting single changes into two parts.
+  std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+      if (can_merge_a) {
+        // Combine into one change
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    merged.push_back(std::move(diff));
+  }
+
+  return merged;
+}
+
+void PrintDifferences(
+    const std::vector<uint64_t>& original, const std::vector<uint64_t>& modified,
+    std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>>& diffs) {
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  std::vector<uint64_t> first = {1, 2, 3, 4, 5};
+  std::vector<uint64_t> second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({2, 3}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  std::vector<uint64_t> first = {1, 2, 3, 4, 5, 6, 7};
+  std::vector<uint64_t> second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({2, 3}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, std::vector<uint64_t>({5}));
+  ASSERT_EQ(diffs[1].second, std::vector<uint64_t>({10}));
+
+  ASSERT_EQ(diffs[2].first, std::vector<uint64_t>({7}));
+  ASSERT_EQ(diffs[2].second, std::vector<uint64_t>({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  std::vector<uint64_t> first = {1, 2, 3};
+  std::vector<uint64_t> second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  std::vector<uint64_t> first = {};
+  std::vector<uint64_t> second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  std::vector<uint64_t> first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  std::vector<uint64_t> second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  std::vector<uint64_t> first = {2169, 1976, 2180, 2147, 1934, 1772,
+                                 1914, 2075, 2154, 1940, 1934, 1970};
+  std::vector<uint64_t> second = {2169, 1976, 2180, 2147, 2265, 1804,
+                                  1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  std::vector<uint64_t> original = {445, 312, 393, 401, 410, 138, 558, 457};
+  std::vector<uint64_t> modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({401}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertUpdateCase(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const std::vector<uint64_t>& original,
+                      const std::vector<uint64_t>& modified, uint8_t n_modifications) {
+  auto diffs = FindDifferences(original, modified);
+  if (diffs.size() > n_modifications) {
+    PrintDifferences(original, modified, diffs);
+  }
+  ASSERT_LE(diffs.size(), n_modifications);
+
+  for (const auto& diff : diffs) {
+    if (!::arrow::is_list_like(dtype->id())) {
+      uint64_t left_sum = 0, right_sum = 0;
+      for (const auto& val : diff.first) left_sum += val;
+      for (const auto& val : diff.second) right_sum += val;
+      ASSERT_EQ(left_sum, right_sum);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original == modified);
+  }
+}
+
+void AssertDeleteCase(const std::shared_ptr<::arrow::DataType>& dtype,

Review Comment:
   Same here and other non-trivial assertion utilities below.



##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, uint64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal128Builder, ::arrow::Decimal128(val % max_value))
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal256Builder, ::arrow::Decimal256(val % max_value))
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      GENERATE_CASE_BODY(::arrow::FixedSizeBinaryBuilder,
+                         std::string("bin_") + std::to_string(val).substr(0, size - 4))
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, uint64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(const std::shared_ptr<Table>& table,
+                                                   uint64_t min_chunk_size,
+                                                   uint64_t max_chunk_size,
+                                                   bool enable_dictionary = false,
+
+                                                   int64_t row_group_size = 1024 * 1024) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_cdc()
+      ->cdc_size_range(min_chunk_size, max_chunk_size)
+      ->cdc_norm_factor(0);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+struct PageSizes {
+  std::vector<uint64_t> lengths;
+  std::vector<uint64_t> sizes;
+};
+
+PageSizes GetColumnPageSizes(const std::shared_ptr<Buffer>& data, int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  PageSizes result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        result.sizes.push_back(data_page->size());
+        result.lengths.push_back(data_page->num_values());
+      }
+    }
+  }
+
+  return result;
+}
+
+Result<PageSizes> WriteAndGetPageSizes(const std::shared_ptr<Table>& table,
+                                       uint64_t min_chunk_size, uint64_t max_chunk_size,
+                                       bool enable_dictionary = false,
+                                       int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnPageSizes(buffer, column_index);
+}
+
+void AssertAllBetween(const std::vector<uint64_t>& values, uint64_t min, uint64_t max,
+                      bool expect_dictionary_fallback = false) {
+  // expect the last chunk since it is not guaranteed to be within the range
+  if (expect_dictionary_fallback) {
+    // if dictionary encoding is enabled, the writer can fallback to plain
+    // encoding splitting within a content defined chunk, so we can't
+    // guarantee that all chunks are within the range in this case, but we
+    // know that there can be at most 2 pages smaller than the min_chunk_size
+    size_t smaller_count = 0;
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      if (values[i] < min) {
+        smaller_count++;
+      } else {
+        ASSERT_LE(values[i], max);
+      }
+    }
+    ASSERT_LE(smaller_count, 2);
+  } else {
+    for (size_t i = 0; i < values.size() - 1; i++) {
+      ASSERT_GE(values[i], min);
+      ASSERT_LE(values[i], max);
+    }
+  }
+  ASSERT_LE(values.back(), max);
+}
+
+std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>> FindDifferences(
+    const std::vector<uint64_t>& first, const std::vector<uint64_t>& second) {
+  // Compute LCS table.
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack to get common indices.
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      i--;
+    } else {
+      j--;
+    }
+  }
+  std::reverse(common.begin(), common.end());
+
+  // Build raw differences.
+  std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Merge adjacent diffs if one side is empty in the first diff and the other side
+  // is empty in the next diff, to avoid splitting single changes into two parts.
+  std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+      if (can_merge_a) {
+        // Combine into one change
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    merged.push_back(std::move(diff));
+  }
+
+  return merged;
+}
+
+void PrintDifferences(
+    const std::vector<uint64_t>& original, const std::vector<uint64_t>& modified,
+    std::vector<std::pair<std::vector<uint64_t>, std::vector<uint64_t>>>& diffs) {
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  std::vector<uint64_t> first = {1, 2, 3, 4, 5};
+  std::vector<uint64_t> second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({2, 3}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  std::vector<uint64_t> first = {1, 2, 3, 4, 5, 6, 7};
+  std::vector<uint64_t> second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({2, 3}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, std::vector<uint64_t>({5}));
+  ASSERT_EQ(diffs[1].second, std::vector<uint64_t>({10}));
+
+  ASSERT_EQ(diffs[2].first, std::vector<uint64_t>({7}));
+  ASSERT_EQ(diffs[2].second, std::vector<uint64_t>({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  std::vector<uint64_t> first = {1, 2, 3};
+  std::vector<uint64_t> second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  std::vector<uint64_t> first = {};
+  std::vector<uint64_t> second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  std::vector<uint64_t> first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  std::vector<uint64_t> second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  std::vector<uint64_t> first = {2169, 1976, 2180, 2147, 1934, 1772,
+                                 1914, 2075, 2154, 1940, 1934, 1970};
+  std::vector<uint64_t> second = {2169, 1976, 2180, 2147, 2265, 1804,
+                                  1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  std::vector<uint64_t> original = {445, 312, 393, 401, 410, 138, 558, 457};
+  std::vector<uint64_t> modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, std::vector<uint64_t>({401}));
+  ASSERT_EQ(diffs[0].second, std::vector<uint64_t>({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertUpdateCase(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const std::vector<uint64_t>& original,
+                      const std::vector<uint64_t>& modified, uint8_t n_modifications) {
+  auto diffs = FindDifferences(original, modified);
+  if (diffs.size() > n_modifications) {
+    PrintDifferences(original, modified, diffs);
+  }
+  ASSERT_LE(diffs.size(), n_modifications);
+
+  for (const auto& diff : diffs) {
+    if (!::arrow::is_list_like(dtype->id())) {
+      uint64_t left_sum = 0, right_sum = 0;
+      for (const auto& val : diff.first) left_sum += val;
+      for (const auto& val : diff.second) right_sum += val;
+      ASSERT_EQ(left_sum, right_sum);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original == modified);
+  }
+}
+
+void AssertDeleteCase(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const std::vector<uint64_t>& original,
+                      const std::vector<uint64_t>& modified, uint8_t n_modifications,
+                      uint64_t edit_length) {
+  auto diffs = FindDifferences(original, modified);
+  if (diffs.size() != n_modifications) {
+    PrintDifferences(original, modified, diffs);
+  }
+  ASSERT_EQ(diffs.size(), n_modifications);
+
+  for (const auto& diff : diffs) {
+    if (!::arrow::is_list_like(dtype->id())) {
+      uint64_t left_sum = 0, right_sum = 0;
+      for (const auto& val : diff.first) left_sum += val;
+      for (const auto& val : diff.second) right_sum += val;
+      ASSERT_EQ(left_sum, right_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+}
+
+void AssertInsertCase(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const std::vector<uint64_t>& original,
+                      const std::vector<uint64_t>& modified, uint8_t n_modifications,
+                      uint64_t edit_length) {
+  auto diffs = FindDifferences(original, modified);
+  if (diffs.size() != n_modifications) {
+    PrintDifferences(original, modified, diffs);
+  }
+  ASSERT_EQ(diffs.size(), n_modifications);
+
+  for (const auto& diff : diffs) {
+    if (!::arrow::is_list_like(dtype->id())) {
+      uint64_t left_sum = 0, right_sum = 0;
+      for (const auto& val : diff.first) left_sum += val;
+      for (const auto& val : diff.second) right_sum += val;
+      ASSERT_EQ(left_sum + edit_length, right_sum);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+}
+
+void AssertAppendCase(const std::vector<uint64_t>& original,
+                      const std::vector<uint64_t>& modified) {
+  ASSERT_GE(modified.size(), original.size());
+  for (size_t i = 0; i < original.size() - 1; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  ASSERT_GT(modified[original.size() - 1], original.back());
+}
+
+uint64_t ElementCount(uint64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      PageSizes base_result, PageSizes modified_result, bool nullable,
+                      bool enable_dictionary, uint64_t min_chunk_size,
+                      uint64_t max_chunk_size) {
+  max_chunk_size = static_cast<uint64_t>(max_chunk_size * 1.2);
+  if (::arrow::is_fixed_width(dtype->id())) {
+    auto min_length = ElementCount(min_chunk_size, dtype->byte_width(), nullable);
+    auto max_length = ElementCount(max_chunk_size, dtype->byte_width(), nullable);
+    AssertAllBetween(base_result.lengths, min_length, max_length,
+                     /*expect_dictionary_fallback=*/enable_dictionary);
+    AssertAllBetween(modified_result.lengths, min_length, max_length,
+                     /*expect_dictionary_fallback=*/enable_dictionary);
+  } else if (::arrow::is_base_binary_like(dtype->id()) && !nullable &&
+             !enable_dictionary) {
+    AssertAllBetween(base_result.sizes, min_chunk_size, max_chunk_size);
+    AssertAllBetween(modified_result.sizes, min_chunk_size, max_chunk_size);
+  }
+}
+
+constexpr uint64_t kMinChunkSize = 8 * 1024;
+constexpr uint64_t kMaxChunkSize = 32 * 1024;
+constexpr uint64_t kPartSize = 128 * 1024;
+constexpr uint64_t kEditSize = 128;
+
+class TestColumnCDC : public ::testing::TestWithParam<
+                          std::tuple<std::shared_ptr<::arrow::DataType>, bool, size_t>> {

Review Comment:
   Can the `std::tuple` be replace with a well-named struct?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1983523442


##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}

Review Comment:
   Removing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1983528187


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {

Review Comment:
   Added the script.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1989650770


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.

Review Comment:
   Is this the approach from the FastCDC paper or is it specific to this PR?



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"

Review Comment:
   Nit: please leave empty line between include groups
   ```suggestion
   
   #include "arrow/array.h"
   ```



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather

Review Comment:
   There might be a missing word here? "one with a lower and one with a probability of matching"



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size

Review Comment:
   Hmm, if we make the mask smaller (less 1s), then the distribution will be closer to the min_size, no?



##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage system (CAS)
+/// which splits the bytes stream into content defined blobs. The CAS system will
+/// calculate a unique identifier for each blob, then store the blob in a key-value store.
+/// If the same blob is encountered again, the system can refer to the hash instead of
+/// physically storing the blob again. In the example above, the CAS system would store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks.
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data
+///   Deduplication
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+/// - Git is for Data (chunk size normalization used here is described in section 6.2.1):
+///   https://www.cidrdb.org/cidr2023/papers/p43-low.pdf
+class ContentDefinedChunker {
+ public:
+  /// Create a new ContentDefinedChunker instance
+  ///
+  /// @param level_info Information about definition and repetition levels
+  /// @param size_range Min/max chunk size as pair<min_size, max_size>, the chunker will
+  ///                   attempt to uniformly distribute the chunks between these extremes.
+  /// @param norm_factor Normalization factor to center the chunk size around the average
+  ///                    size more aggressively. By increasing the normalization factor,
+  ///                    probability of finding a chunk boundary increases.
+  ContentDefinedChunker(const LevelInfo& level_info,
+                        std::pair<uint64_t, uint64_t> size_range,
+                        uint8_t norm_factor = 0);
+
+  /// Get the chunk boundaries for the given column data
+  ///
+  /// @param def_levels Definition levels
+  /// @param rep_levels Repetition levels
+  /// @param num_levels Number of levels
+  /// @param values Column values as an Arrow array
+  /// @return Vector of Chunk objects representing the chunk boundaries
+  const std::vector<Chunk> GetBoundaries(const int16_t* def_levels,
+                                         const int16_t* rep_levels, int64_t num_levels,
+                                         const ::arrow::Array& values);
+
+ private:
+  // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+  // true if the hash matches the mask.
+  template <typename T>
+  void Roll(const T value);
+

Review Comment:
   Yes, it makes the header file cleaner and it lets us include less also.



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             int64_t min_size, int64_t max_size,
+                                             int8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(min_size),
+      max_size_(max_size),
+      hash_mask_(GetMask(min_size, max_size, norm_factor)) {
+  if (min_size_ < 0) {
+    throw ParquetException("min_size must be non-negative");
+  }
+  if (max_size_ < 0) {
+    throw ParquetException("max_size must be non-negative");
+  }
+  if (min_size_ > max_size_) {
+    throw ParquetException("min_size must be less than or equal to max_size");
+  }
+}
+
+void ContentDefinedChunker::Roll(const bool value) {
+  if (chunk_size_++ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+  has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+}
+
+template <int ByteWidth>
+void ContentDefinedChunker::Roll(const uint8_t* value) {
+  chunk_size_ += ByteWidth;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (size_t i = 0; i < ByteWidth; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T* value) {
+  return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+}
+
+void ContentDefinedChunker::Roll(const uint8_t* value, int64_t length) {
+  chunk_size_ += length;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (auto i = 0; i < length; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, note that we don't reset the
+    // rolling hash state here, so the next NeedNewChunk() call will continue from the
+    // current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename RollFunc>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,

Review Comment:
   `const` on the return value is pointless.



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             int64_t min_size, int64_t max_size,
+                                             int8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(min_size),
+      max_size_(max_size),
+      hash_mask_(GetMask(min_size, max_size, norm_factor)) {
+  if (min_size_ < 0) {
+    throw ParquetException("min_size must be non-negative");
+  }
+  if (max_size_ < 0) {
+    throw ParquetException("max_size must be non-negative");
+  }

Review Comment:
   We should also mandate that they are non-zero, IMHO.



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));

Review Comment:
   Can use `arrow::bit_util::Log2` instead which will avoid a costly logarithm.



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             int64_t min_size, int64_t max_size,
+                                             int8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(min_size),
+      max_size_(max_size),
+      hash_mask_(GetMask(min_size, max_size, norm_factor)) {
+  if (min_size_ < 0) {
+    throw ParquetException("min_size must be non-negative");
+  }
+  if (max_size_ < 0) {
+    throw ParquetException("max_size must be non-negative");
+  }
+  if (min_size_ > max_size_) {
+    throw ParquetException("min_size must be less than or equal to max_size");
+  }
+}
+
+void ContentDefinedChunker::Roll(const bool value) {
+  if (chunk_size_++ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+  has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+}
+
+template <int ByteWidth>
+void ContentDefinedChunker::Roll(const uint8_t* value) {
+  chunk_size_ += ByteWidth;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (size_t i = 0; i < ByteWidth; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T* value) {
+  return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+}
+
+void ContentDefinedChunker::Roll(const uint8_t* value, int64_t length) {
+  chunk_size_ += length;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (auto i = 0; i < length; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, note that we don't reset the
+    // rolling hash state here, so the next NeedNewChunk() call will continue from the
+    // current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename RollFunc>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const RollFunc& RollValue) {
+  std::vector<Chunk> chunks;
+  int64_t offset;
+  int64_t prev_offset = 0;
+  int64_t prev_value_offset = 0;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    for (offset = 0; offset < num_levels; ++offset) {
+      RollValue(offset);
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int16_t def_level;
+    for (int64_t offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+
+      Roll(&def_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(offset);
+      }
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else {
+    // nested data with nulls
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t value_offset = 0;
+
+    for (offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+      rep_level = rep_levels[offset];
+
+      Roll(&def_level);
+      Roll(&rep_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(value_offset);
+      }
+
+      if ((rep_level == 0) && NeedNewChunk()) {
+        // if we are at a record boundary and need a new chunk, we create a new chunk
+        auto levels_to_write = offset - prev_offset;
+        if (levels_to_write > 0) {
+          chunks.emplace_back(prev_offset, prev_value_offset, levels_to_write);
+          prev_offset = offset;
+          prev_value_offset = value_offset;
+        }
+      }
+      if (def_level >= level_info_.repeated_ancestor_def_level) {
+        // we only increment the value offset if we have a leaf value
+        ++value_offset;
+      }
+    }
+  }
+
+  // add the last chunk if we have any levels left
+  if (prev_offset < num_levels) {
+    chunks.emplace_back(prev_offset, prev_value_offset, num_levels - prev_offset);
+  }
+  return chunks;
+}
+
+#define FIXED_WIDTH_CASE(ByteWidth)                                       \
+  {                                                                       \
+    const auto raw_values = values.data()->GetValues<uint8_t>(1);         \
+    return Calculate(def_levels, rep_levels, num_levels, [&](int64_t i) { \
+      return Roll<ByteWidth>(raw_values + i * ByteWidth);                 \
+    });                                                                   \
+  }
+
+#define BINARY_LIKE_CASE(ArrayType)                                       \
+  {                                                                       \
+    const auto& array = static_cast<const ArrayType&>(values);            \
+    const uint8_t* value;                                                 \
+    ArrayType::offset_type length;                                        \
+    return Calculate(def_levels, rep_levels, num_levels, [&](int64_t i) { \
+      value = array.GetValue(i, &length);                                 \
+      Roll(value, length);                                                \
+    });                                                                   \
+  }
+
+const std::vector<Chunk> ContentDefinedChunker::GetBoundaries(
+    const int16_t* def_levels, const int16_t* rep_levels, int64_t num_levels,
+    const ::arrow::Array& values) {
+  auto type_id = values.type()->id();
+  switch (type_id) {
+    case ::arrow::Type::NA: {
+      return Calculate(def_levels, rep_levels, num_levels, [](int64_t) {});
+    }
+    case ::arrow::Type::BOOL: {
+      const auto& bool_array = static_cast<const ::arrow::BooleanArray&>(values);
+      return Calculate(def_levels, rep_levels, num_levels,
+                       [&](int64_t i) { return Roll(bool_array.Value(i)); });
+    }
+    case ::arrow::Type::INT8:
+    case ::arrow::Type::UINT8:
+      FIXED_WIDTH_CASE(1)
+    case ::arrow::Type::INT16:
+    case ::arrow::Type::UINT16:
+    case ::arrow::Type::HALF_FLOAT:
+      FIXED_WIDTH_CASE(2)
+    case ::arrow::Type::INT32:
+    case ::arrow::Type::UINT32:
+    case ::arrow::Type::FLOAT:
+    case ::arrow::Type::DATE32:
+    case ::arrow::Type::TIME32:
+      FIXED_WIDTH_CASE(4)
+    case ::arrow::Type::INT64:
+    case ::arrow::Type::UINT64:
+    case ::arrow::Type::DOUBLE:
+    case ::arrow::Type::DATE64:
+    case ::arrow::Type::TIME64:
+    case ::arrow::Type::TIMESTAMP:
+    case ::arrow::Type::DURATION:
+      FIXED_WIDTH_CASE(8)
+    case ::arrow::Type::DECIMAL128:

Review Comment:
   Should also decimal32 and decimal64.



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             int64_t min_size, int64_t max_size,
+                                             int8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(min_size),
+      max_size_(max_size),
+      hash_mask_(GetMask(min_size, max_size, norm_factor)) {
+  if (min_size_ < 0) {
+    throw ParquetException("min_size must be non-negative");
+  }
+  if (max_size_ < 0) {
+    throw ParquetException("max_size must be non-negative");
+  }
+  if (min_size_ > max_size_) {
+    throw ParquetException("min_size must be less than or equal to max_size");
+  }
+}
+
+void ContentDefinedChunker::Roll(const bool value) {
+  if (chunk_size_++ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+  has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+}
+
+template <int ByteWidth>
+void ContentDefinedChunker::Roll(const uint8_t* value) {
+  chunk_size_ += ByteWidth;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (size_t i = 0; i < ByteWidth; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T* value) {
+  return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+}
+
+void ContentDefinedChunker::Roll(const uint8_t* value, int64_t length) {
+  chunk_size_ += length;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (auto i = 0; i < length; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, note that we don't reset the
+    // rolling hash state here, so the next NeedNewChunk() call will continue from the
+    // current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename RollFunc>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const RollFunc& RollValue) {
+  std::vector<Chunk> chunks;
+  int64_t offset;
+  int64_t prev_offset = 0;
+  int64_t prev_value_offset = 0;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    for (offset = 0; offset < num_levels; ++offset) {
+      RollValue(offset);
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int16_t def_level;
+    for (int64_t offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+
+      Roll(&def_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(offset);
+      }
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else {
+    // nested data with nulls
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t value_offset = 0;
+
+    for (offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+      rep_level = rep_levels[offset];
+
+      Roll(&def_level);
+      Roll(&rep_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(value_offset);
+      }
+
+      if ((rep_level == 0) && NeedNewChunk()) {
+        // if we are at a record boundary and need a new chunk, we create a new chunk
+        auto levels_to_write = offset - prev_offset;
+        if (levels_to_write > 0) {
+          chunks.emplace_back(prev_offset, prev_value_offset, levels_to_write);
+          prev_offset = offset;
+          prev_value_offset = value_offset;
+        }
+      }
+      if (def_level >= level_info_.repeated_ancestor_def_level) {
+        // we only increment the value offset if we have a leaf value
+        ++value_offset;
+      }
+    }
+  }
+
+  // add the last chunk if we have any levels left
+  if (prev_offset < num_levels) {
+    chunks.emplace_back(prev_offset, prev_value_offset, num_levels - prev_offset);
+  }
+  return chunks;
+}
+
+#define FIXED_WIDTH_CASE(ByteWidth)                                       \
+  {                                                                       \
+    const auto raw_values = values.data()->GetValues<uint8_t>(1);         \
+    return Calculate(def_levels, rep_levels, num_levels, [&](int64_t i) { \
+      return Roll<ByteWidth>(raw_values + i * ByteWidth);                 \
+    });                                                                   \
+  }
+
+#define BINARY_LIKE_CASE(ArrayType)                                       \
+  {                                                                       \
+    const auto& array = static_cast<const ArrayType&>(values);            \
+    const uint8_t* value;                                                 \
+    ArrayType::offset_type length;                                        \
+    return Calculate(def_levels, rep_levels, num_levels, [&](int64_t i) { \
+      value = array.GetValue(i, &length);                                 \
+      Roll(value, length);                                                \
+    });                                                                   \
+  }
+
+const std::vector<Chunk> ContentDefinedChunker::GetBoundaries(
+    const int16_t* def_levels, const int16_t* rep_levels, int64_t num_levels,
+    const ::arrow::Array& values) {
+  auto type_id = values.type()->id();
+  switch (type_id) {

Review Comment:
   I think it would be easier, nicer and more future-proof to use `VisitType` with a generic lambda and some Calculate helpers. Something like this:
   ```c++
   std::vector<Chunk> CalculateBoolean(
       const int16_t* def_levels, const int16_t* rep_levels,
       int64_t num_levels, const Array& values) {
     const auto& bool_array = checked_cast<const ::arrow::BooleanArray&>(values);
     return Calculate(def_levels, rep_levels, num_levels,
                      [&](int64_t i) { return Roll(bool_array.Value(i)); });
   }
   
   template <int kByteWidth>
   std::vector<Chunk> CalculateFixedWidth(
       const int16_t* def_levels, const int16_t* rep_levels,
       int64_t num_levels, const Array& values) {
     const uint8_t* raw_values = values.data()->GetValues(1, 0) + values.offset() * kByteWidth;
     return Calculate(def_levels, rep_levels, num_levels,
                      [&](int64_t i) { return Roll<kByteWidth>(&raw_values[i * kByteWidth]); });
   }
   
   template <typename ArrayType>
   std::vector<Chunk> CalculateBinaryLike(
       const int16_t* def_levels, const int16_t* rep_levels,
       int64_t num_levels, const Array& values) {
     const auto& binary_array = checked_cast<const ArrayType&>(values);
     const uint8_t* raw_values = values.data()->GetValues(1, 0) + values.offset() * kByteWidth;
     return Calculate(def_levels, rep_levels, num_levels,
                      [&](int64_t i) { return Roll(binary_array.GetView(i)); });
   }
   
   std::vector<Chunk> ContentDefinedChunker::GetBoundaries(
       const int16_t* def_levels, const int16_t* rep_levels, int64_t num_levels,
       const ::arrow::Array& values) {
     auto handle_type = [&](auto&& type) {
       using Type = std::decay_t<decltype(type)>;
       if constexpr (::arrow::is_boolean(Type::type_id)) {
         return CalculateBoolean(def_levels, rep_levels, num_levels, values);
       }
       else if constexpr (::arrow::is_primitive(Type::type_id)) {
         using c_type = typename ::arrow::type_traits<Type>::CType;
         return CalculateFixedWidth<sizeof(c_type)>(def_levels, rep_levels, num_levels, values);
       }
       else if constexpr (::arrow::is_binary_like(Type::type_id)) {
         return CalculateBinaryLike<::arrow::BinaryArray>(def_levels, rep_levels, num_levels, values);
       }
       else if constexpr (::arrow::is_large_binary_like(Type::type_id)) {
         return CalculateBinaryLike<::arrow::LargeBinaryArray>(def_levels, rep_levels, num_levels, values);
       }
       /* else etc. */
     };
     return VisitType(*values.type(), handle_type);
   }
   ```



##########
cpp/src/parquet/chunker_internal_codegen.py:
##########
@@ -0,0 +1,116 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Produce the given number gearhash tables for rolling hash calculations.
+
+Each table consists of 256 64-bit integer values and by default 8 tables are
+produced. The tables are written to a header file that can be included in the
+C++ code.
+
+The generated numbers are deterministic "random" numbers created by MD5 hashing
+a fixed seed and the table index. This ensures that the tables are the same
+across different runs and platforms. The function of generating the numbers is
+less important as long as they have sufficiently uniform distribution.
+
+Reference implementations:
+- https://github.com/Borelset/destor/blob/master/src/chunking/fascdc_chunking.c
+- https://github.com/nlfiedler/fastcdc-rs/blob/master/examples/table64.rs
+
+Usage:
+    python chunker_internal_codegen.py [ntables]
+
+    ntables: Number of gearhash tables to generate (default 8), the
+             the C++ implementation expects 8 tables so this should not be
+             changed unless the C++ code is also updated.
+
+    The generated header file is written to ./chunker_internal_generated.h
+"""
+
+import hashlib
+import pathlib
+import sys
+from io import StringIO
+
+
+template = """\
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+#include <cstdint>
+
+namespace parquet::internal {{
+
+constexpr uint64_t kGearhashTable[8][256] = {{
+{content}}};
+
+}}  // namespace parquet::internal
+"""
+
+
+def generate_hash(n: int, seed: int):
+    """Produce predictable hash values for a given seed and n using MD5."""
+    value = bytes([seed] * 64 + [n] * 64)

Review Comment:
   Do you have an idea why `* 64` is being used? Is there an explanation somewhere?



##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}

Review Comment:
   You can probably just `push_back({record_level_offset, record_value_offset, levels_to_write})`?



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             int64_t min_size, int64_t max_size,
+                                             int8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(min_size),
+      max_size_(max_size),
+      hash_mask_(GetMask(min_size, max_size, norm_factor)) {
+  if (min_size_ < 0) {
+    throw ParquetException("min_size must be non-negative");
+  }
+  if (max_size_ < 0) {
+    throw ParquetException("max_size must be non-negative");
+  }
+  if (min_size_ > max_size_) {
+    throw ParquetException("min_size must be less than or equal to max_size");
+  }
+}
+
+void ContentDefinedChunker::Roll(const bool value) {
+  if (chunk_size_++ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+  has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+}
+
+template <int ByteWidth>
+void ContentDefinedChunker::Roll(const uint8_t* value) {
+  chunk_size_ += ByteWidth;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (size_t i = 0; i < ByteWidth; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T* value) {
+  return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+}
+
+void ContentDefinedChunker::Roll(const uint8_t* value, int64_t length) {
+  chunk_size_ += length;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (auto i = 0; i < length; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, note that we don't reset the
+    // rolling hash state here, so the next NeedNewChunk() call will continue from the
+    // current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename RollFunc>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const RollFunc& RollValue) {
+  std::vector<Chunk> chunks;
+  int64_t offset;
+  int64_t prev_offset = 0;
+  int64_t prev_value_offset = 0;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    for (offset = 0; offset < num_levels; ++offset) {
+      RollValue(offset);
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int16_t def_level;
+    for (int64_t offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+
+      Roll(&def_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(offset);
+      }
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else {
+    // nested data with nulls
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t value_offset = 0;
+
+    for (offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+      rep_level = rep_levels[offset];
+
+      Roll(&def_level);
+      Roll(&rep_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(value_offset);
+      }
+
+      if ((rep_level == 0) && NeedNewChunk()) {
+        // if we are at a record boundary and need a new chunk, we create a new chunk
+        auto levels_to_write = offset - prev_offset;
+        if (levels_to_write > 0) {
+          chunks.emplace_back(prev_offset, prev_value_offset, levels_to_write);
+          prev_offset = offset;
+          prev_value_offset = value_offset;
+        }
+      }
+      if (def_level >= level_info_.repeated_ancestor_def_level) {
+        // we only increment the value offset if we have a leaf value
+        ++value_offset;
+      }
+    }
+  }
+
+  // add the last chunk if we have any levels left
+  if (prev_offset < num_levels) {
+    chunks.emplace_back(prev_offset, prev_value_offset, num_levels - prev_offset);
+  }
+  return chunks;
+}
+
+#define FIXED_WIDTH_CASE(ByteWidth)                                       \
+  {                                                                       \
+    const auto raw_values = values.data()->GetValues<uint8_t>(1);         \

Review Comment:
   You need to take the array offset into account.



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,250 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_hashtable.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEARHASH_TABLE[nth_run_][bytes[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);

Review Comment:
   I'm not sure I understand your answer to be honest. You're going to chunk at _value_ boundaries, not at byte boundaries, so why update `has_matched_` at every byte?



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             int64_t min_size, int64_t max_size,
+                                             int8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(min_size),
+      max_size_(max_size),
+      hash_mask_(GetMask(min_size, max_size, norm_factor)) {
+  if (min_size_ < 0) {
+    throw ParquetException("min_size must be non-negative");
+  }
+  if (max_size_ < 0) {
+    throw ParquetException("max_size must be non-negative");
+  }
+  if (min_size_ > max_size_) {
+    throw ParquetException("min_size must be less than or equal to max_size");
+  }
+}
+
+void ContentDefinedChunker::Roll(const bool value) {
+  if (chunk_size_++ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+  has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+}
+
+template <int ByteWidth>
+void ContentDefinedChunker::Roll(const uint8_t* value) {
+  chunk_size_ += ByteWidth;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (size_t i = 0; i < ByteWidth; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T* value) {
+  return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+}
+
+void ContentDefinedChunker::Roll(const uint8_t* value, int64_t length) {
+  chunk_size_ += length;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (auto i = 0; i < length; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and

Review Comment:
   Is there a reference to the "exponential distribution" claim?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1989704534


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             int64_t min_size, int64_t max_size,
+                                             int8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(min_size),
+      max_size_(max_size),
+      hash_mask_(GetMask(min_size, max_size, norm_factor)) {
+  if (min_size_ < 0) {
+    throw ParquetException("min_size must be non-negative");
+  }
+  if (max_size_ < 0) {
+    throw ParquetException("max_size must be non-negative");
+  }
+  if (min_size_ > max_size_) {
+    throw ParquetException("min_size must be less than or equal to max_size");
+  }
+}
+
+void ContentDefinedChunker::Roll(const bool value) {
+  if (chunk_size_++ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+  has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+}
+
+template <int ByteWidth>
+void ContentDefinedChunker::Roll(const uint8_t* value) {
+  chunk_size_ += ByteWidth;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (size_t i = 0; i < ByteWidth; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T* value) {
+  return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+}
+
+void ContentDefinedChunker::Roll(const uint8_t* value, int64_t length) {
+  chunk_size_ += length;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (auto i = 0; i < length; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, note that we don't reset the
+    // rolling hash state here, so the next NeedNewChunk() call will continue from the
+    // current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename RollFunc>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const RollFunc& RollValue) {
+  std::vector<Chunk> chunks;
+  int64_t offset;
+  int64_t prev_offset = 0;
+  int64_t prev_value_offset = 0;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    for (offset = 0; offset < num_levels; ++offset) {
+      RollValue(offset);
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int16_t def_level;
+    for (int64_t offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+
+      Roll(&def_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(offset);
+      }
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else {
+    // nested data with nulls
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t value_offset = 0;
+
+    for (offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+      rep_level = rep_levels[offset];
+
+      Roll(&def_level);
+      Roll(&rep_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(value_offset);
+      }
+
+      if ((rep_level == 0) && NeedNewChunk()) {
+        // if we are at a record boundary and need a new chunk, we create a new chunk
+        auto levels_to_write = offset - prev_offset;
+        if (levels_to_write > 0) {
+          chunks.emplace_back(prev_offset, prev_value_offset, levels_to_write);
+          prev_offset = offset;
+          prev_value_offset = value_offset;
+        }
+      }
+      if (def_level >= level_info_.repeated_ancestor_def_level) {
+        // we only increment the value offset if we have a leaf value
+        ++value_offset;
+      }
+    }
+  }
+
+  // add the last chunk if we have any levels left
+  if (prev_offset < num_levels) {
+    chunks.emplace_back(prev_offset, prev_value_offset, num_levels - prev_offset);
+  }
+  return chunks;
+}
+
+#define FIXED_WIDTH_CASE(ByteWidth)                                       \
+  {                                                                       \
+    const auto raw_values = values.data()->GetValues<uint8_t>(1);         \
+    return Calculate(def_levels, rep_levels, num_levels, [&](int64_t i) { \
+      return Roll<ByteWidth>(raw_values + i * ByteWidth);                 \
+    });                                                                   \
+  }
+
+#define BINARY_LIKE_CASE(ArrayType)                                       \
+  {                                                                       \
+    const auto& array = static_cast<const ArrayType&>(values);            \
+    const uint8_t* value;                                                 \
+    ArrayType::offset_type length;                                        \
+    return Calculate(def_levels, rep_levels, num_levels, [&](int64_t i) { \
+      value = array.GetValue(i, &length);                                 \
+      Roll(value, length);                                                \
+    });                                                                   \
+  }
+
+const std::vector<Chunk> ContentDefinedChunker::GetBoundaries(
+    const int16_t* def_levels, const int16_t* rep_levels, int64_t num_levels,
+    const ::arrow::Array& values) {
+  auto type_id = values.type()->id();
+  switch (type_id) {

Review Comment:
   I think it would be easier, nicer and more future-proof to use `VisitType` with a generic lambda and some Calculate helpers. Something like this:
   ```c++
   std::vector<Chunk> CalculateBoolean(
       const int16_t* def_levels, const int16_t* rep_levels,
       int64_t num_levels, const Array& values) {
     const auto& bool_array = checked_cast<const ::arrow::BooleanArray&>(values);
     return Calculate(def_levels, rep_levels, num_levels,
                      [&](int64_t i) { return Roll(bool_array.Value(i)); });
   }
   
   template <int kByteWidth>
   std::vector<Chunk> CalculateFixedWidth(
       const int16_t* def_levels, const int16_t* rep_levels,
       int64_t num_levels, const Array& values) {
     const uint8_t* raw_values = values.data()->GetValues(1, 0) + values.offset() * kByteWidth;
     return Calculate(def_levels, rep_levels, num_levels,
                      [&](int64_t i) { return Roll<kByteWidth>(&raw_values[i * kByteWidth]); });
   }
   
   template <typename ArrayType>
   std::vector<Chunk> CalculateBinaryLike(
       const int16_t* def_levels, const int16_t* rep_levels,
       int64_t num_levels, const Array& values) {
     const auto& binary_array = checked_cast<const ArrayType&>(values);
     return Calculate(def_levels, rep_levels, num_levels,
                      [&](int64_t i) { return Roll(binary_array.GetView(i)); });
   }
   
   std::vector<Chunk> ContentDefinedChunker::GetBoundaries(
       const int16_t* def_levels, const int16_t* rep_levels, int64_t num_levels,
       const ::arrow::Array& values) {
     auto handle_type = [&](auto&& type) {
       using Type = std::decay_t<decltype(type)>;
       if constexpr (::arrow::is_boolean(Type::type_id)) {
         return CalculateBoolean(def_levels, rep_levels, num_levels, values);
       }
       else if constexpr (::arrow::is_primitive(Type::type_id)) {
         using c_type = typename ::arrow::type_traits<Type>::CType;
         return CalculateFixedWidth<sizeof(c_type)>(def_levels, rep_levels, num_levels, values);
       }
       else if constexpr (::arrow::is_binary_like(Type::type_id)) {
         return CalculateBinaryLike<::arrow::BinaryArray>(def_levels, rep_levels, num_levels, values);
       }
       else if constexpr (::arrow::is_large_binary_like(Type::type_id)) {
         return CalculateBinaryLike<::arrow::LargeBinaryArray>(def_levels, rep_levels, num_levels, values);
       }
       /* else etc. */
     };
     return VisitType(*values.type(), handle_type);
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1968422591


##########
cpp/src/parquet/CMakeLists.txt:
##########
@@ -390,6 +391,7 @@ add_parquet_test(reader-test
 
 add_parquet_test(writer-test
                  SOURCES
+                 column_chunker_test.cc

Review Comment:
   It should possibly moved to a separate test unit. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086140180


##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,144 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <vector>
+
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  // The start offset of this chunk inside the given levels
+  int64_t level_offset;
+  // The start offset of this chunk inside the given values array
+  int64_t value_offset;
+  // The length of the chunk in levels
+  int64_t levels_to_write;
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which is contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...

Review Comment:
   Yes, it is trying to highlight the behavior since I would need a lot more values for a reproducible example.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2875493770

   @github-actions crossbow submit -g cpp


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086668026


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1689 @@
+// Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   `if defined(ADDRESS_SANITIZER) || defined(ARROW_VALGRIND)`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou merged PR #45360:
URL: https://github.com/apache/arrow/pull/45360


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2083098045


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int norm_factor) {
+  if (min_chunk_size < 0) {

Review Comment:
   Yes it can, effectively disabling "cut-point skipping". Updated the error message to non-negative.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2085252152


##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage system (CAS)
+/// which splits the bytes stream into content defined blobs. The CAS system will
+/// calculate a unique identifier for each blob, then store the blob in a key-value store.
+/// If the same blob is encountered again, the system can refer to the hash instead of
+/// physically storing the blob again. In the example above, the CAS system would store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.
+/// While the deduplication is performed by the CAS system, the parquet chunker makes it
+/// possible to efficiently deduplicate the data by consistently dividing the data into
+/// chunks.
+///
+/// Implementation details:
+///
+/// Only the parquet writer must be aware of the content defined chunking, the reader
+/// doesn't need to know about it. Each parquet column writer holds a
+/// ContentDefinedChunker instance depending on the writer's properties. The chunker's
+/// state is maintained across the entire column without being reset between pages and row
+/// groups.
+///
+/// The chunker receives the record shredded column data (def_levels, rep_levels, values)
+/// and goes over the (def_level, rep_level, value) triplets one by one while adjusting
+/// the column-global rolling hash based on the triplet. Whenever the rolling hash matches
+/// a predefined mask, the chunker creates a new chunk. The chunker returns a vector of
+/// Chunk objects that represent the boundaries of the chunks.
+/// Note that the boundaries are deterministically calculated exclusively based on the
+/// data itself, so the same data will always produce the same chunks - given the same
+/// chunker configuration.
+///
+/// References:
+/// - FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data
+///   Deduplication
+///   https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
+/// - Git is for Data (chunk size normalization used here is described in section 6.2.1):
+///   https://www.cidrdb.org/cidr2023/papers/p43-low.pdf
+class ContentDefinedChunker {
+ public:
+  /// Create a new ContentDefinedChunker instance
+  ///
+  /// @param level_info Information about definition and repetition levels
+  /// @param size_range Min/max chunk size as pair<min_size, max_size>, the chunker will
+  ///                   attempt to uniformly distribute the chunks between these extremes.
+  /// @param norm_factor Normalization factor to center the chunk size around the average
+  ///                    size more aggressively. By increasing the normalization factor,
+  ///                    probability of finding a chunk boundary increases.

Review Comment:
   Renamed it to `norm_level`. We can adjust or make it clearer in a follow-up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086133191


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,412 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+namespace {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_level parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_level Normalization level (default 0)
+// @return The mask used to compare against the rolling hash
+uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size, int norm_level) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be non-negative");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes; also divide by the number of gearhash tables to have a
+  // a more gaussian-like distribution
+  int64_t target_size = (avg_chunk_size - min_chunk_size) / kNumGearhashTables;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // a user defined `norm_level` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_level we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_level is 0 by
+  // default
+  int effective_bits = mask_bits - norm_level;
+
+  if (effective_bits < 1 || effective_bits > 63) {
+    throw ParquetException(
+        "The number of bits in the CDC mask must be between 1 and 63, got " +
+        std::to_string(effective_bits));
+  } else {
+    // create the mask by setting the top bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+}  // namespace
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int norm_level)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_level)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the

Review Comment:
   I don't think we should add `ARROW:` in front of those keys. There is nothing Arrow-specific here (as in: CDC is not related to the Arrow memory format).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1946145115


##########
cpp/src/parquet/column_chunker.h:
##########
@@ -0,0 +1,784 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+using arrow::internal::checked_cast;
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+const int64_t AVG_LEN = 1024 * 1024;
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t avg_len, size_t adjustement_level) {
+  size_t mask_bits = static_cast<size_t>(std::ceil(std::log2(avg_len)));
+  return (1ULL << (mask_bits - adjustement_level)) - 1;
+}
+
+// rename it since it is not FastCDC anymore
+class FastCDC {
+ public:
+  FastCDC(const LevelInfo& level_info, uint64_t avg_len, uint8_t granurality_level = 5)
+      : level_info_(level_info),
+        avg_len_(avg_len == 0 ? AVG_LEN : avg_len),
+        min_len_(static_cast<uint64_t>(avg_len_ * 0.6)),

Review Comment:
   Maybe we should expose min and max chunk size again, these factors try to normalize the chunk size.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1999485538


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             int64_t min_size, int64_t max_size,
+                                             int8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(min_size),
+      max_size_(max_size),
+      hash_mask_(GetMask(min_size, max_size, norm_factor)) {
+  if (min_size_ < 0) {
+    throw ParquetException("min_size must be non-negative");
+  }
+  if (max_size_ < 0) {
+    throw ParquetException("max_size must be non-negative");
+  }

Review Comment:
   I am adding some extra validation about the mask size as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2005178288


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             int64_t min_size, int64_t max_size,
+                                             int8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(min_size),
+      max_size_(max_size),
+      hash_mask_(GetMask(min_size, max_size, norm_factor)) {
+  if (min_size_ < 0) {
+    throw ParquetException("min_size must be non-negative");
+  }
+  if (max_size_ < 0) {
+    throw ParquetException("max_size must be non-negative");
+  }
+  if (min_size_ > max_size_) {
+    throw ParquetException("min_size must be less than or equal to max_size");
+  }
+}
+
+void ContentDefinedChunker::Roll(const bool value) {
+  if (chunk_size_++ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+  has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+}
+
+template <int ByteWidth>
+void ContentDefinedChunker::Roll(const uint8_t* value) {
+  chunk_size_ += ByteWidth;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (size_t i = 0; i < ByteWidth; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T* value) {
+  return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+}
+
+void ContentDefinedChunker::Roll(const uint8_t* value, int64_t length) {
+  chunk_size_ += length;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (auto i = 0; i < length; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, note that we don't reset the
+    // rolling hash state here, so the next NeedNewChunk() call will continue from the
+    // current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename RollFunc>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const RollFunc& RollValue) {
+  std::vector<Chunk> chunks;
+  int64_t offset;
+  int64_t prev_offset = 0;
+  int64_t prev_value_offset = 0;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    for (offset = 0; offset < num_levels; ++offset) {
+      RollValue(offset);
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int16_t def_level;
+    for (int64_t offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+
+      Roll(&def_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(offset);
+      }
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else {
+    // nested data with nulls
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t value_offset = 0;
+
+    for (offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+      rep_level = rep_levels[offset];
+
+      Roll(&def_level);
+      Roll(&rep_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(value_offset);
+      }
+
+      if ((rep_level == 0) && NeedNewChunk()) {
+        // if we are at a record boundary and need a new chunk, we create a new chunk
+        auto levels_to_write = offset - prev_offset;
+        if (levels_to_write > 0) {
+          chunks.emplace_back(prev_offset, prev_value_offset, levels_to_write);
+          prev_offset = offset;
+          prev_value_offset = value_offset;
+        }
+      }
+      if (def_level >= level_info_.repeated_ancestor_def_level) {
+        // we only increment the value offset if we have a leaf value
+        ++value_offset;
+      }
+    }
+  }
+
+  // add the last chunk if we have any levels left
+  if (prev_offset < num_levels) {
+    chunks.emplace_back(prev_offset, prev_value_offset, num_levels - prev_offset);
+  }
+  return chunks;
+}
+
+#define FIXED_WIDTH_CASE(ByteWidth)                                       \
+  {                                                                       \
+    const auto raw_values = values.data()->GetValues<uint8_t>(1);         \

Review Comment:
   Added test for sliced tables. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2005288118


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \
+  {                                                    \
+    BUILDER_TYPE builder(type, default_memory_pool()); \
+    if (nullable) {                                    \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        if (val % 10 == 0) {                           \
+          RETURN_NOT_OK(builder.AppendNull());         \
+        } else {                                       \
+          RETURN_NOT_OK(builder.Append(VALUE_EXPR));   \
+        }                                              \
+      }                                                \
+    } else {                                           \
+      for (int64_t i = 0; i < length; ++i) {           \
+        uint64_t val = hash(seed, i);                  \
+        RETURN_NOT_OK(builder.Append(VALUE_EXPR));     \
+      }                                                \
+    }                                                  \
+    std::shared_ptr<Array> array;                      \
+    RETURN_NOT_OK(builder.Finish(&array));             \
+    RETURN_NOT_OK(array->ValidateFull());              \
+    return array;                                      \
+  }
+
+// Macro to generate a case for a given scalar type.
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR) \
+  case ::arrow::Type::TYPE_ID: {                         \
+    GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)         \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, uint64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal128Builder, ::arrow::Decimal128(val % max_value))
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      GENERATE_CASE_BODY(::arrow::Decimal256Builder, ::arrow::Decimal256(val % max_value))
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      GENERATE_CASE_BODY(::arrow::FixedSizeBinaryBuilder,
+                         std::string("bin_") + std::to_string(val).substr(0, size - 4))
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, uint64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(const std::shared_ptr<Table>& table,
+                                                   uint64_t min_chunk_size,
+                                                   uint64_t max_chunk_size,
+                                                   bool enable_dictionary = false,
+
+                                                   int64_t row_group_size = 1024 * 1024) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_cdc()
+      ->cdc_size_range(min_chunk_size, max_chunk_size)
+      ->cdc_norm_factor(0);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }

Review Comment:
   Updated, testing that the row group has dictionary page in case of `enable_dictionary` (except for boolean). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027848020


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);

Review Comment:
   This is only used from the UpdateOnce and UpdateTwice test cases where the edit(s) don't change the length of the original array, only update the value (this happens to be true for the list types as well), so we can be fairly strict here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027849232


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);

Review Comment:
   Added a comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027903459


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};

Review Comment:
   Yes, added. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2739940873

   I think I managed to address most review comments, I have two minor things left:
   - do more validation on the CDC chunker parameters
   - place some information about the used chunking in the metadata


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kou (via GitHub)" <gi...@apache.org>]

kou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1973016269


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }

Review Comment:
   I'm not sure whether this is a critical path or not but we may be able to eliminate this branch by `has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0)`. This may improve performance.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1968436601


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {

Review Comment:
   Here with dictionary encoding disabled and `norm_factor=1` (instead of the default 0):
   
   ![newplot-10](https://github.com/user-attachments/assets/092b4d8f-2e8e-4e7c-a609-2d4ac8f4d3d1)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1968424740


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {

Review Comment:
   These constants are generated by a python script, copy-pasting is simple but I am open to suggestions. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1981281943


##########
cpp/src/parquet/column_chunker.h:
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+
+namespace internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  int64_t level_offset;
+  int64_t value_offset;
+  int64_t levels_to_write;
+
+  Chunk(int64_t level_offset, int64_t value_offset, int64_t levels_to_write)
+      : level_offset(level_offset),
+        value_offset(value_offset),
+        levels_to_write(levels_to_write) {}
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which are contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...
+///
+/// Then the parquet file is being uploaded to a content addressable storage systems (CAS)
+/// which split the bytes stream into content defined blobs. The CAS system will calculate
+/// a unique identifier for each blob, then store the blob in a key-value store. If the
+/// same blob is encountered again, the system can refer to the hash instead of physically
+/// storing the blob again. In the example above, the CAS system would phiysically store
+/// Page1, Page2, Page3, and Page4 only once and the required metadata to reassemble the
+/// files.

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1976493402


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+bool ContentDefinedChunker::Check() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last Check() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling
+    // hash state here, so the next Check() call will continue from the current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename T>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const T& leaf_array) {
+  std::vector<Chunk> result;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(def_levels[offset]);
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else {
+    // nested data with nulls
+    bool has_leaf_value;
+    bool is_record_boundary;
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t level_offset = 0;
+    int64_t value_offset = 0;
+    int64_t record_level_offset = 0;
+    int64_t record_value_offset = 0;
+
+    while (level_offset < num_levels) {
+      def_level = def_levels[level_offset];
+      rep_level = rep_levels[level_offset];
+
+      has_leaf_value = def_level >= level_info_.repeated_ancestor_def_level;
+      is_record_boundary = rep_level == 0;
+
+      Roll(def_level);
+      Roll(rep_level);
+      if (has_leaf_value) {
+        Roll(leaf_array.GetView(value_offset));
+      }
+
+      if (is_record_boundary && Check()) {
+        auto levels_to_write = level_offset - record_level_offset;
+        if (levels_to_write > 0) {
+          result.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+          record_level_offset = level_offset;
+          record_value_offset = value_offset;
+        }
+      }
+
+      ++level_offset;
+      if (has_leaf_value) {
+        ++value_offset;
+      }
+    }
+
+    auto levels_to_write = num_levels - record_level_offset;
+    if (levels_to_write > 0) {
+      result.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+    }
+  }
+
+  return result;
+}
+
+#define PRIMITIVE_CASE(TYPE_ID, ArrowType)               \
+  case ::arrow::Type::TYPE_ID:                           \
+    return Calculate(def_levels, rep_levels, num_levels, \
+                     static_cast<const ::arrow::ArrowType##Array&>(values));
+
+const ::arrow::Result<std::vector<Chunk>> ContentDefinedChunker::GetBoundaries(
+    const int16_t* def_levels, const int16_t* rep_levels, int64_t num_levels,
+    const ::arrow::Array& values) {
+  auto type_id = values.type()->id();
+  switch (type_id) {

Review Comment:
   > I tried, but it makes the implementation way more complicated.
   
   



##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {
+    {// seed = 0
+     0xf09f35a563783945, 0x0dcc5b3bc5ae410a, 0x63f1ea8d22554270, 0xfbe5ee7bd05a7b61,
+     0x3f692ed5e9934aba, 0xaab3755952250eb8, 0xdefb168dc2888fa5, 0x501b36f7c77a7d47,
+     0xd2fff45d1989642d, 0x80217c1c600e30a6, 0xb9469ee2e43df7ac, 0x3654b76a61999706,
+     0x6ea73dfe5de0c6b6, 0xdfd662e1937a589d, 0x0dbe0cc74b188a68, 0xde45f4e6d73ffc6f,
+     0xcdf7a7759e70d87e, 0x5d6a951b8d38c310, 0xdc9423c3813fcf2c, 0x25dc2976e167ffce,
+     0xc2555baa1d031c84, 0x115bc3f2230a3ab6, 0xd4b10260f350bede, 0xdfd3501ab447d723,
+     0x022e79217edaf167, 0x1635e2255c5a7526, 0xa0a750350cc77102, 0xc027133e05d39f56,
+     0xd949459779cf0387, 0xb92f1464f5c688c2, 0xd9ac5f3e8b42f2f3, 0xdf02bb6f5ecaac21,
+     0x8156f988fac7bfa4, 0xe4580f97bede2ec8, 0x44fe7d17a76fca32, 0x885f59bd54c2014c,
+     0x435e63ec655ffae9, 0x5ebc51930967b1f1, 0x5428c2084ac29e47, 0x9465938fec30e36b,
+     0xc7cb3de4977772cd, 0x15692d7c201e8c3a, 0x505ee65cdc4b17f4, 0x7d9839a0a7aead6b,
+     0xeef5f5b6a0105291, 0x76c2fb232ce7f5bf, 0x5c13893c1c3ff3a9, 0x65b6b547d4442f98,
+     0xb8ad7487c8c96fce, 0x906bcf51c99974f8, 0x2f56e48bb943a48c, 0xbc9ab109f82d3a44,
+     0xcd5160cdc8c7e735, 0xbe9acb9df3427732, 0x386b91d477d7fade, 0x36be463621dd5af2,
+     0xcbe6a2faffd627a8, 0x9c8fd528463a2f5a, 0xb9b88c6bb802b184, 0xb414b4e665c597c7,
+     0xbedb142568209556, 0x5360d81c25429dce, 0x63a69a960a952f37, 0xc900d63899e1b503,
+     0x1abc63a8b37c7728, 0xa8b3a8b6409080eb, 0x495e391f662959f6, 0xdf1e136f3e12229b,
+     0x33d5fc526b0dd38d, 0x321221ae2abfac63, 0x7fde18351fda7395, 0xed79fe5c3a6aa4c3,
+     0x2dd6965a4867d8d4, 0x54813ca20fe8799b, 0x5d59ea6456465c39, 0x0de0c294d1936b81,
+     0x4aaf0755002c588c, 0x3530a1857ad04c6d, 0xb8a64f4ce184442b, 0xe0def10bceedfa17,
+     0x46e38d0a443757ec, 0x9795a1c645ee16d7, 0x7e531def245eac8a, 0x683b25c43a0716cf,
+     0x884583d372da219d, 0x5b06b62c910416e5, 0x54b6902fbebd3dbe, 0x931198d40a761a75,
+     0xead7d8e830013590, 0x80b4d5dc99bfaced, 0xf98272c8108a1ad2, 0x1adce054289a0ec6,
+     0x7d53a1143c56b465, 0x497fbe4f00c92b52, 0x525e4cc2e81ebd69, 0xc94478e0d5508ff6,
+     0xb8a5da83c196d07c, 0x7667a921b65b0603, 0xf236fabbdefe6cd1, 0x53da978d19a92b98,
+     0xc604f6e97087124d, 0x2cbd27221924b094, 0x65cd1102c985b1d2, 0x08c0755dc1a97eb4,
+     0x5e0419e921c0fef1, 0x282d2c1196f84a29, 0xe21117fcfc5793f7, 0xcf4e985dc38e6c2e,
+     0xd521f4f264d55616, 0xde69b04c485f2a10, 0x59410e245305178a, 0xceab1d477c943601,
+     0xa9805732d71ee5e9, 0x054cd443896974f6, 0xf2b517717a423a3e, 0x09517937fa9fac95,
+     0x4938233e9ca871e3, 0x9132cbaf56f83ec0, 0x4703421ed1dd027d, 0xfd9933f4e6f1ec4e,
+     0xf237c7fded2274a8, 0xdf4616efe68cd7b4, 0x5e46de0f39f0a380, 0x3d41e0c6d8e095b0,
+     0xc5272f8a5bb2df09, 0x68aa78e8301fb964, 0xbf5b5b52c8e32ae0, 0xbf28ed3df74bdcf7,
+     0xd6198f64c833815a, 0x8cd99d2974267544, 0xd90560ea4465ff2c, 0x571d65ad7ad59261,
+     0x309453518baa367a, 0xa60538377bc79fb2, 0xace515da1ab4183c, 0xf56d3c8d891d1c5b,
+     0x5b0d8370b59def49, 0x775866ce7c83c762, 0x3d76085695c8e18a, 0xba064d1a9af1b114,
+     0xc84ef7cd7b98b521, 0x90b9231681c2bc37, 0x37e2b13e6f585b6b, 0x1d0a34e55e0f369f,
+     0x86bb8019cf41447c, 0x4b95c6ef55b3f71f, 0x3b6ed1660732b310, 0x617eee603d137f21,
+     0xf4f6278b464f3bbc, 0xdfb763b720da205a, 0x353478899b871cb7, 0xe45fbbff574cc41e,
+     0x1a94b60847907d72, 0xb10eef051eff67a5, 0xf0e012ec6a284d40, 0xcc1cd1a11b926d7c,
+     0xcf9d9c5453e19cad, 0x270febcc0fc0e86b, 0xd6567568778b781e, 0x7323b98965eeb46b,
+     0xccecd374567086ff, 0xef7b44bfc497a704, 0xebc479c051a9f0a5, 0xc9b7410e3e00a235,
+     0x1d084f7ecdf83dab, 0xc8a9a97e33ba8ba3, 0x8c75318f5b2350d6, 0xaa3cd5d0c684bdda,
+     0xa81125fe0901bedf, 0xf7bcd76020edfc93, 0x834ee4c12e75874f, 0xb2bb8a7beb44fa14,
+     0x32cd26f50a4f4e4d, 0x0fc5817ca55d959a, 0xd6e4ae2e3ae10718, 0x074abdcceb8d6e38,
+     0xc0cc5f4f9b3a9c43, 0x1115d364363595b2, 0x69861db2eb19f2e8, 0x59b8d804cf92bc67,
+     0x9bac9785e5e4b863, 0x7fa0e17a41869561, 0x10d3c9633f0c709c, 0x534a03deee6bc44a,
+     0x73b1f7201257f581, 0x46fd6a11e2e0706b, 0x494abb554946e67a, 0xb5d6da317864dc8e,
+     0x402ded9238f39687, 0xd8fa37d2cbd6d290, 0xcc818293fcb06791, 0x6482ab344806cd4d,
+     0x0956e6ee9d8eb60b, 0x01fee622d8465ac8, 0xae7ece370cbd9c35, 0x7ff09e937a177279,
+     0xa2c29ee7a33ca5f1, 0x990e8dbee083923b, 0x4a819b72f610863a, 0xddecfad79d3f08be,
+     0x627372480fac20a7, 0x802154d6eca2db4c, 0x8fcf02e42f805e55, 0x040a911ff8cea977,
+     0xbb544485bc64d0d4, 0xaddde1aeb406d0fb, 0xf6b35fae23dce66f, 0xc07a9fb3645d2f9b,
+     0xccd113907e9c0fed, 0xd17af369984fd213, 0x9223823c59a083e7, 0xe19d475606b81013,
+     0xe181ac116a90e57a, 0x71f7b6258c6def4c, 0x2246f34b45964f7c, 0xd74aedaea2d31751,
+     0xb1add86e5dd305d1, 0xeb9ba881f16d6471, 0xef7600e036f5c6ff, 0x1d50bc9735b8fb85,
+     0xe63942bd1f3e2969, 0x9241ba9f8b3f4e72, 0xee8bb2bca07d35b6, 0x55cd55dab522654e,
+     0x94d0cfa7c1a6845d, 0x02f9845d559884c3, 0x8ce70ea21063b560, 0xd70998028ef08b74,
+     0xdfdb5bbee310876b, 0x4e21b2e348256d16, 0xde007a981c13debc, 0xe51950cbbddabfdd,
+     0xd223301dbe9957c1, 0x084b8634cc2cce4b, 0x90e551378aa9d70c, 0x833b533ac633e448,
+     0x7891e232882da57f, 0xa1bf26f0163ce2b3, 0xf33a0171eb9c68d5, 0x2e7de18ca69b3fa2,
+     0x666fd6f175619199, 0x1239d37edb5feb9f, 0xfa9fc9382e61ff5c, 0x3ca4ad427e3c126f,
+     0x37c6dd4c2c31ae6e, 0x1f1bacb619d427b2, 0x7dd09f5d10759afe, 0xc8d941432327d733,
+     0x2b389ba25e1d43a7, 0xa4e3030c3740ff21, 0xcc56dae13fd37463, 0x2481457c175b560f,
+     0x9deb35bde77c5c41, 0x847aa6ea5549a0c3, 0xcde01bb48b6e7f02, 0x15a28844e64cb211},
+    {// seed = 1
+     0xecfcba92fe5691a3, 0x71377799fea34699, 0xb284c9096fa614e5, 0x54534170f40de6c8,
+     0xbbd804d45884fba3, 0x44929a896388c8a1, 0x79b712508e0fa3b1, 0xeb53ab280af31054,
+     0x351ea23a6319da7a, 0x2fbe55d9819d85a2, 0x34f4b6568dcd28b1, 0x8c94ea5e5d82967a,
+     0x09068d333a46d3c5, 0x762ad4f64cb73381, 0xd5c6db5ef0e22640, 0x36d8ab5a36175680,
+     0xd41fe333cdc3525a, 0xa1f51dbdf20ce781, 0x1410a95e786c8be6, 0x96b7499a670c2b41,
+     0x3912e1037835d893, 0x272c5bd83e1e9115, 0x2ea7f91cad82a0d6, 0xcd10e85662ce9931,
+     0xedad49be8d5e8b74, 0x7ccd8fe0f37d12bc, 0xfac0482005eed593, 0x4513991681f6c8b0,
+     0x2804d612eb0ad37d, 0x7cca9e8412b81d34, 0x85ffd6707192b7b8, 0xea0560aeea954411,
+     0x0122d28226102bba, 0xf51c47cdbd22fdd1, 0x3707d851183ff17c, 0xaef5a1465f3e902d,
+     0xbcb38c2d8736a04f, 0x4025317e864bef15, 0x8d3f66d86e1ea58f, 0xc16759a3d97ed79a,
+     0x1c62abdc0659f2f5, 0x23b3eb4e699bd28f, 0x5083c4fceed3ccaf, 0xa65bf34562cc989c,
+     0xaa5865932fd79064, 0xf24d08d268c24593, 0x7fbd00a215196999, 0x7812cd366d752964,
+     0x62e8dcb27ef3d945, 0xf08b7984e1b946dc, 0x547d23ad9a5c1dcf, 0x496b1fb249b27fb7,
+     0xcd692e1db5f3b3ba, 0x41931e39f1e1bc61, 0x286c6a7d7edae82b, 0x17ef6638b6c4ca6e,
+     0x609beb5a2576a934, 0xcc5e16fe4a69b83c, 0xbbd14d08b078fc24, 0x2a617680f481cb94,
+     0x81dbbd5f86e6d039, 0xeb8205e1fc8ecc3c, 0xe5e3bb576faa8042, 0x5d6f1eb9d9df01b5,
+     0x9a47b8739c10fb44, 0x398a7caad7ea7696, 0x9c0fc1d7c46adde6, 0x67cd6de0a51978a6,
+     0x68ccc4b77a21cca4, 0x1e067066b82f415c, 0xf7ddade6535e1819, 0xf2185c884291751b,
+     0xc322b7381fcbe34f, 0x242f593e88290b9b, 0x8e11ccc0ea5e84a3, 0x40e3a2e3346db8a2,
+     0xf18bfc3ad2931a2c, 0x2468397394b00144, 0xeae199cce14e6817, 0x05b462686c75a1ae,
+     0xda096cb859c51673, 0xd87aeb967a906bef, 0xaabc74493cb02fe6, 0x74d48fc2e7da143e,
+     0x6ec1c8fed3f2c1fd, 0xe01e0704b463f18e, 0xc3d88a4d3a8056e4, 0xd01ae0ffab6c8f3f,
+     0x881ba052620ae7c7, 0xcea033aef0a823a5, 0x8d2cad91d83df1e3, 0x18746d205e66dbe9,
+     0x3061f8e58d046650, 0xd819c59f0ce2cf8b, 0x144e89e93635e870, 0x3415e88279b21651,
+     0xd6f7ab944b86c3fa, 0x45f1dd15d0f67bdc, 0xbf0d97c7f4fa24f4, 0x34a7de520a57fcd2,
+     0x4ba86fda03e9e2bc, 0xa7995265a025b552, 0x698f6819d5f51cf7, 0xd07dbe9d8a156981,
+     0x2683945373857fc1, 0x116f8a84f96167de, 0x8bc832bd85595ebf, 0xb206519d74fdfafa,
+     0xde9519b2e9b5cc5f, 0x16fdd6f2da1d8163, 0x7ba32bd48ef56f11, 0x6f4e4d7ee8b29717,
+     0xd31576dde7468aad, 0x023bb08848676045, 0xf6dcc083178160b7, 0x42035f426250e683,
+     0x343732993cfed89f, 0x0640a870a22d3d58, 0x65cff80b53b4ae6a, 0x27996fa17ab05215,
+     0xfd5db01401b21a04, 0x894508784bc1673c, 0x5bfcf43a2380e27d, 0x4cd6dcc2715583b7,
+     0xa43b3763e7d4c902, 0x6da83e12ef0c1257, 0xfe80a602b0335aff, 0x293a7d8f4ff344de,
+     0xb4ae7c2b8956bf5a, 0x6b45432d38254b4d, 0xd086acbdf15d9455, 0xa4d19e43f41ea87b,
+     0xf01f13ba4bb87fbf, 0xca582cf301a299ff, 0x0ddad3d45298fa7d, 0x0646a130459c3999,
+     0xc08e3af3747e2cee, 0xfc7db8aa9ed67295, 0x783b329e7bd79d5f, 0x732dbc607957af7b,
+     0x8e446ac19fb26555, 0xff1dfa4d61dc89a5, 0xb6fbc46bd8d011d8, 0x185147ec5779f0d7,
+     0x6eb2cf6149a5380f, 0xb0e773df803a1eae, 0xc07706c5519bfce5, 0xc35abcf54fa95f14,
+     0x40a01d99a38608ea, 0x776dcd6f603c277f, 0x6ae12389b1d6d0bb, 0x8bd981448df92bb9,
+     0x426a6a7ca21a2c16, 0x87efd5b71c1bad26, 0x71fb7fc4cd41de48, 0xdd9033c45619d463,
+     0x40eaab322654cef7, 0xe077fffed6f3e3a2, 0x375a4dbef9384447, 0x2066b009d2c4a100,
+     0xeca4a5794a068447, 0x2128f64bddf341a1, 0x738b4bb1be90bd61, 0x433772cf3813d52e,
+     0x9540c88add8e4474, 0x0b6d5decd21d3519, 0x654ead966745642d, 0xe1bfb03c3b4bdb4c,
+     0x0b977a9937515b1f, 0x0a4587509ef63870, 0xe89f0de1d9cfd44a, 0x23a91390272e7f68,
+     0xd92defbc9096b8d8, 0x004db87174612539, 0xc88ecaabdd1a71f1, 0x050de38393073346,
+     0x8af1426d7964e038, 0xf352c4fef8ad5c87, 0x6f26bc7408e26548, 0x0d41543fd9bf3084,
+     0xfc4e07553a840fc6, 0x5ef117de86a555a9, 0x1f11c42dffb5ae1b, 0x4147648f07490fa5,
+     0x09b35fd7671b21aa, 0x1453b14f7ccca481, 0x944f6fcce4c9b2ba, 0x5b08dd2e3583dc06,
+     0xe0220df78dc9c22d, 0x1c200b9506cbf666, 0x8a0b7465eadb523b, 0xfbcb43a91a1e2d80,
+     0xe697f44be3c36a58, 0x2f8a8e48fb7e350d, 0x7baba71b8920d55f, 0x10edc0216105bc96,
+     0x52db07c79d7a7a63, 0x1916e8cef9452ac3, 0x5cbbbf21f867b6cc, 0xadd583365a690a4b,
+     0x4e4ca2c8bffc2fdb, 0xf5fe3416d2eebcfe, 0x839af8b85e452476, 0x8496c0c54ad44e16,
+     0x6c46f1ecad4482bf, 0xb794cad76ae18715, 0x67b762eec7c62985, 0x52dc9e68df5b3a53,
+     0x0cc7e444b422a5f9, 0xadbfe90841c112b0, 0xfe37b136f0ca5c34, 0xcfe9e47948a8d73e,
+     0xee90572b86a30d91, 0x549e72d8262830aa, 0x3361564b469f32c6, 0x1e6eba9e0d2648e2,
+     0x5f8e2b2ac5fcb4eb, 0xe4224fa5f71f7cc6, 0x7357a9230c76757b, 0xcad70f74aaf6b702,
+     0xeef28ced23894cc2, 0x753fdd3352aefd68, 0x1fed6ba90bbeb9d2, 0x05316f4ab4034b4b,
+     0x3396df022b9f63d6, 0x82d7125a7cfd0935, 0x3519a71caf1f87f0, 0xd1dfb7a5cc3974be,
+     0xbfae40ecbdbbcc2a, 0x152c11778e08dd54, 0x4a96566a6c848554, 0x3a84d621c340cdd7,
+     0xfd47aa1887e2fb03, 0xa63cae94b2f1d099, 0xed61783f3e5b75e0, 0xefd44864106019be,
+     0x145ff78b80b081aa, 0x34670e5fcea9230e, 0x876ef976328db371, 0x4221f3a5269942a6,
+     0x95315cbd85c648f4, 0x3ca344dc7c3b1600, 0x38421ea39ff28780, 0x31dbeee967c0435c,
+     0x27437c3e268402e7, 0xdd0cf8343312a654, 0x965ab9dad1d8aa29, 0xf871706dd3e23509,
+     0xce23d06c7a25e699, 0x1b37d59382b27589, 0x3407f004723d6324, 0x56efb69cdb5deaa1,
+     0xf46cdd2b9fd604e0, 0xcad3ca79fdac69bd, 0x7252802a574e63cb, 0xc281fb8acc6ec1d3},
+    {// seed = 2
+     0xdd16cb672ba6979c, 0x3954eaa9ec41ae41, 0x52cb802771d2966d, 0xf57ed8eb0d0294f2,
+     0x768be23c71da2219, 0x6131e22d95a84ad3, 0xd849e4e49bb15842, 0x18e8e5c4978cf00d,
+     0x3af5e5867ce1f9bd, 0x06c75a9fffe83d63, 0xe8de75a00b58a065, 0x0a773251bc0d755a,
+     0x629dc21e54548329, 0x2a168f5e5a883e70, 0x33547375f0996c86, 0xdfcb4c7680451322,
+     0x55c1ecaaaa57e397, 0x4546c346c24f5a31, 0x6f8f0401dfabc86c, 0x7760d2d36ee340b4,
+     0xf6448e48bdeb229d, 0xba70e1633b4dba65, 0x069cda561e273054, 0xa010b6a84aebf340,
+     0x5c23b8229eee34b6, 0xea63c926d90153af, 0x7d7de27b3e43ec1b, 0xea119541eddc3491,
+     0xf1259daeddfc724c, 0x2873ca9a67730647, 0xa1e7710dade32607, 0x758de030b61d43fd,
+     0xd2c9bcbfa475edb4, 0x18ade47bb8a0aa29, 0xf7a74af0ff1aea88, 0x6f8873274a987162,
+     0x6963e8d876f4d282, 0xd435d4fe448c6c5b, 0x93ec80ba404cafff, 0xcf90d24c509e41e7,
+     0x5f0fc8a62923e36e, 0x9224878fe458f3a4, 0xd9a039edf1945bcd, 0x0877d1892c288441,
+     0x75205491f4b4740b, 0x30f9d2d523a9085b, 0x4b7f4029fa097c99, 0x170bb013745709d4,
+     0x7087af537f11ef2e, 0x28c62b88e08fc464, 0x84bbcb3e0bb56271, 0x485a4b099165c681,
+     0x357c63357caa9292, 0x819eb7d1aee2d27e, 0xdaa759eb9c0f8c9d, 0x42cdc36729cc3db5,
+     0x9489aa852eddbb06, 0x8161e4f85a84e6d4, 0xa964863fdad3eb29, 0xcc095ddbce1a6702,
+     0x3ecfadbb8dc2ce58, 0x971316509b95a231, 0xc8f484d1dbc38427, 0xae9c510c463574c0,
+     0xdf2b31179600c21a, 0x440de87bada4dfa3, 0xbd8d30f3f6fb7522, 0x84e6d7f678a0e2d0,
+     0x0ec4d74323e15975, 0xf6947610dad6d9ab, 0x73a55a95d73fe3a5, 0x3e5f623024d37eda,
+     0x8d99a728d95d9344, 0x8b82a7956c4acdc4, 0x7faeaea4385b27f6, 0x540625ff4aa2ff21,
+     0x4aa43b3ebd92ce2b, 0x899646a6df2da807, 0x49225115780942d7, 0xe16606636af89525,
+     0xb980bcf893888e33, 0xf9ed57695291b0d8, 0x5c6dd14464619afa, 0x50606d69b733d4f3,
+     0x7fb1af465b990f97, 0x3fab2634c8bbd936, 0x556da6168838b902, 0x0f15975902a30e1f,
+     0xb29d782ae9e1991f, 0xae00e26ff8f7e739, 0xd3da86458bb292d5, 0x4528ee0afb27e4ce,
+     0x49882d5ba49fabad, 0x7e873b6a7cf875ee, 0x777edd535113c912, 0x94ed05e7ff149594,
+     0x0b8f95fc4211df43, 0x9135c2b42426fef2, 0x411e6c2b47307073, 0x503207d1af0c8cf8,
+     0xd76f8619059f9a79, 0x64d24617855dee45, 0xf7bc7a877923196a, 0xd6cc42ed6a65be79,
+     0xe3912ff09d4fc574, 0x4192d03b2bc2460a, 0xa0dcc37dad98af85, 0xfc59049b2a5818a4,
+     0x2128bae90a5b975f, 0xbe7067ca05ea3294, 0x5bab7e7753064c4f, 0x42cbf0949ef88443,
+     0x564df4bbd017492c, 0xf2c2eb500cf80564, 0x5b92e67eb00e92af, 0x8c4103eef59c0341,
+     0x83412122b8284998, 0x888daf2da0636b6d, 0x4d54b10303dd07d6, 0x201190e7c1e7b5ed,
+     0x3797510bb53a5771, 0x03f7bc598b570b79, 0xdc1e15d67d94f73e, 0x721e8b499ebe02c1,
+     0x71f954f606d13fa0, 0x0c7a2e408c168bf0, 0x07df2ef14f69c89d, 0xe295096f46b4baaf,
+     0x7a2037916438737e, 0xd1e861aeaf8676ea, 0xb36ebdce368b8108, 0xb7e53b090ddb5d25,
+     0x5a606607b390b1aa, 0x475e52994f4a2471, 0xbcc2038ba55b2078, 0x28b8a6b6c80df694,
+     0xb5f0130ec972c9a2, 0x7a87cd2a93276b54, 0x4d0eec7ecf92d625, 0xac1a8ce16269a42e,
+     0xa4ca0237ca9637b8, 0xd8dc8ff91202b6ff, 0x75b29846799d7678, 0x761b11a5edd9c757,
+     0xf2581db294ef3307, 0xe3173c2b6a48e20f, 0xe46fd7d486d65b3c, 0x1352024303580d1f,
+     0x2d665dae485c1d6d, 0x4e0905c825d74d3b, 0x14ff470c331c229e, 0xbdc656b8613d8805,
+     0x36de38e396345721, 0xaae682c1aa8ff13b, 0x57eb28d7b85a1052, 0xf3145290231d443a,
+     0xd0f68095e23cbe39, 0x67f99b3c2570b33d, 0x54575285f3017a83, 0x9b2f7bb03d836a79,
+     0xa57b209d303367a9, 0x7ccb545dd0939c79, 0x1392b79a37f4716d, 0x6e81bb91a3c79bcd,
+     0x2c2cd80307dddf81, 0xb949e119e2a16cbb, 0x69625382c4c7596f, 0xf19c6d97204fb95c,
+     0x1b2ea42a24b6b05e, 0x8976f83cd43d20ac, 0x7149dd3de44c9872, 0xc79f1ae2d2623059,
+     0xca17a4f143a414e1, 0x66d7a1a21b6f0185, 0xed2c6198fe73f113, 0x16a5f0295cbe06af,
+     0x5f27162e38d98013, 0xf54d9f295bdc0f76, 0x9ba7d562073ef77b, 0xa4a24daaa2cfc571,
+     0x49884cf486da43cd, 0x74c641c0e2148a24, 0xbff9dcbff504c482, 0xf8fc2d9403c837ab,
+     0x6ccc44828af0bb1e, 0xbcf0d69b4c19dfdb, 0x8fe0d962d47abf8f, 0xa65f1d9d5514271d,
+     0x26ff393e62ef6a03, 0xc7153500f283e8fc, 0xea5ed99cdd9d15cd, 0xfc16ac2ba8b48bb7,
+     0xf49694b70041c67a, 0xbd35dd30f5d15f72, 0xcf10ad7385f83f98, 0x709e52e27339cdc2,
+     0xe9505cb3ec893b71, 0x2ffa610e4a229af7, 0x12e1bc774d1f0e52, 0xe301a3bb7eacccc8,
+     0x1fdd3b6dcd877ebf, 0x56a7e8bda59c05aa, 0x99acd421035d6ab4, 0xfd21e401cecd2808,
+     0x9a89d23df8b8d46f, 0x4e26b1f1eb297b9c, 0x9df24d973e1eae07, 0xe6cdc74da62a6318,
+     0xfc360d74df992db0, 0xf4eca0a739514c98, 0x481c515ba9bf5215, 0xce89cce80f5f3022,
+     0xf487a10fc80e4777, 0x235b379a87e41832, 0x76f72e028371f194, 0xd044d4a201325a7d,
+     0x47d8e855e0ffbdde, 0x268ae196fe7334b0, 0x123f2b26db46faa8, 0x11741175b86eb083,
+     0x72ee185a423e6e31, 0x8da113dfe6f6df89, 0x286b72e338bbd548, 0xa922246204973592,
+     0x7237b4f939a6b629, 0x31babda9bedf039a, 0xb2e8f18c6aeec258, 0x0f5f6ce6dd65a45e,
+     0x8f9071a0f23e57d3, 0x71307115ba598423, 0xcbe70264c0e1768c, 0x1c23729f955681a8,
+     0xfbc829099bc2fc24, 0x9619355cbc37d5d6, 0xea694d4e59b59a74, 0xb41cf8d3a7c4f638,
+     0xae1e792df721cd0b, 0x7cd855d28aac11f6, 0xca11ba0efec11238, 0x7c433e554ce261d8,
+     0xe3140366f042b6ba, 0x8a59d68642b3b18c, 0x094fcdd5d7bccac2, 0x9517d80356362c37,
+     0x4a20a9949c6c74e8, 0xc25bcf1699d3b326, 0xa8893f1d1ed2f340, 0x9b58986e0e8a886e,
+     0x29d78c647587ce41, 0x3b210181df471767, 0xd45e8e807627849d, 0x1ec56bc3f2b653e3,
+     0x974ff23068558b00, 0xdb72bdac5d34262c, 0x23225143bb206b57, 0xd0a34cfe027cbb7e},
+    {// seed = 3
+     0x39209fb3eb541043, 0xee0cd3754563088f, 0x36c05fc545bf8abe, 0x842cb6381a9d396b,
+     0xd5059dcb443ce3bf, 0xe92545a8dfa7097e, 0xb9d47558d8049174, 0xc6389e426f4c2fc0,
+     0xd8e0a6e4c0b850d3, 0x7730e54360bd0d0d, 0x6ecb4d4c50d050d5, 0x07a16584d4eb229f,
+     0x13305d05f4a92267, 0xb278ddd75db4baec, 0x32381b774138608f, 0x61fe7a7163948057,
+     0x460c58a9092efee6, 0x553bf895d9b5ff62, 0x899daf2dabfd0189, 0xf388ab9c1c4b6f70,
+     0xd600fe47027ea4cd, 0x16d527ec2b5ef355, 0x5ac1f58ff6908c81, 0xa08d79ff8ee9ffe8,
+     0xc1060a80b7a5e117, 0x14b2c23118c60bda, 0x8cc0defbb890df8f, 0xe29540fd94c6d28b,
+     0xa604f003f82d5b71, 0xa67583d4eb066d18, 0xd62cbd796322b3fc, 0x070cfe244cdcccf3,
+     0x73557c30b3af47e5, 0x2e544e31153a2163, 0x996eef7464d5bead, 0xbc71cb5ab0586cdc,
+     0x0bfcb6c1b517ed69, 0x62b4f1fcc82e8ca0, 0x0edbc68f544965c5, 0x40fa39baa24af412,
+     0xf39aeb2413dab165, 0x17e6013e7afee738, 0x8109bff1c8d42a9d, 0x3cd99863390989b5,
+     0x02021a4cc9c336c8, 0xa06060778cb60aa4, 0xd96591db60bc1e06, 0xd2727175183f4022,
+     0xcdc1f1c5bce3e7ce, 0xb393ccc447872a37, 0xdf6efe63257ead3a, 0x20729d0340dbceb6,
+     0x9f3d2d26fc0ea0d7, 0xf392e0885189bd79, 0xdf2ee01eb212b8b6, 0x6e103a0c0f97e2c3,
+     0x96c604a763bd841b, 0x9fc590c43bba0169, 0xf92dcd5ddc248c40, 0x113a8b54446941dc,
+     0x5943eda146b46bb8, 0xbf657901a36a39a7, 0x5a4e0e7ea6568971, 0xb94c635bae9f9117,
+     0x2626fb65b3a4ef81, 0xa59bfd5478ce97de, 0x79112ba9cc1a1c63, 0xf41f102f002cf39c,
+     0x0a589bcbfb7ff1c8, 0xa1478c53540c4fa1, 0x60d55e72c86dfaca, 0x312e7b6840ea7a39,
+     0x8aae72dcccfe1f75, 0xff2f51f55bf0247a, 0x3c2e4b109edb4a90, 0x5c6d73f6525c7637,
+     0xe49acb04a199f61c, 0x27860642d966df7f, 0x541ce75fb1e21c30, 0xd9fcd6f90806c7cc,
+     0xb87c27bc93a7969b, 0x92f77a1179b8f8dc, 0xb1f29379deb89ed4, 0x7e63ead35808efe7,
+     0x13545183d7fa5420, 0x575f593e34cf029d, 0x27f1199fb07344ae, 0xe67f95f7dc741455,
+     0x49b478b761ab850b, 0xd7bedf794adfc21e, 0xdc788dcd2dda40ae, 0x14673eb9f4d8ad35,
+     0x0cced3c71ecf5eb1, 0xe62d4e6c84471180, 0xdfe1b9e2cb4ada7d, 0x70185a8fce980426,
+     0x0ce2db5e8f9553d6, 0x1fedc57bb37b7264, 0xb9310a2e970b3760, 0x989ff8ab9805e87d,
+     0x0b912d7eb712d9ee, 0x1fe272830379e67c, 0x16e6a73aff4738fb, 0xeed196d98ba43866,
+     0x7088ca12d356cbe2, 0x23539aa43a71eee0, 0xed52f0311fa0f7ad, 0xa12b16233f302eea,
+     0xc477786f0870ecb4, 0xd603674717a93920, 0x4abe0ae17fa62a4c, 0xa18f1ad79e4edc8d,
+     0xc49fe6db967c6981, 0xcc154d7e3c1271e9, 0xdd075d640013c0c0, 0xc026cd797d10922a,
+     0xead7339703f95572, 0x4342f6f11739eb4b, 0x9862f4657d15c197, 0x4f3cb1d4d392f9ff,
+     0xe35bffa018b97d03, 0x600c755031939ad3, 0xb8c6557ffea83abf, 0x14c9e7f2f8a122ea,
+     0x0a2eb9285ee95a7c, 0x8823fec19840c46f, 0x2c4c445c736ed1d0, 0x83181dff233449f1,
+     0x15ed3fca3107bef5, 0x305e9adb688a4c71, 0x7dbef196f68a3e2e, 0x93e47ece3e249187,
+     0x8353c5e890ead93c, 0xea8a7ae66abafdf7, 0xf956dbb6becf7f74, 0x9f37c494fbfdb6e4,
+     0x11c6cbaa2485dd32, 0x206f336fcca11320, 0x9befe9a59135d8fe, 0x5f3ef8b8db92c7db,
+     0xbb305e556ce0ce9a, 0xf26bdafb1305887f, 0xcbf28abe23f08c61, 0x0bc64173b914e00b,
+     0x9168da52e983f54a, 0x6ea41d09c3574a3e, 0x78aa44d4a74459ae, 0x2931422878387bf5,
+     0x018f64a3a92c2d9c, 0x9be43f6752e66b34, 0xae378890decd1152, 0x07325329a1cb7623,
+     0x3b96f4ee3dd9c525, 0x2d6ebcdbe77d61a3, 0x10e32b0e975f510c, 0xffc007b9da959bf9,
+     0x38bf66c6559e5d90, 0xbe22bdf0bf8899fe, 0x87807d7a991632a8, 0x149a0d702816766a,
+     0x026f723db057e9ab, 0xeeecb83625ec6798, 0xcec2ed5984208148, 0xd985a78e97f03c84,
+     0xf96c279e7927b116, 0x99d5027b3204f6e2, 0x13a84878c3d34c55, 0x5cf5ec96229e9676,
+     0x0bc36b07e4f8e289, 0xbed33b80a069914d, 0x2fbfbdd1ff4b9396, 0xab352bb6982da90f,
+     0x154d219e4fa3f62b, 0x4d087512bb6b9be7, 0xc582e31775ee400e, 0x7dadb002ae8c4a4e,
+     0xaae2957375c1aee2, 0x5f36ca643356625b, 0xf87cf8eb76e07fb7, 0x46f432a755e02cc3,
+     0x36087e07aba09642, 0xe5642c1e4ebb9939, 0xb9152d22338eefad, 0xf7ba44278a22cf7f,
+     0xd3b8013502acd838, 0x7761511da6482659, 0xb0857621638e8e50, 0x552eddb4a8b1d5f5,
+     0xc43d9861e812c3ea, 0xd765c2aada47910c, 0x21c935b68f552b19, 0x6256d5641a2b47dc,
+     0xab711d8e6c94bc79, 0xa8d0b91a2a01ab81, 0x5e6d66141e8d632a, 0x7638285124d5d602,
+     0x794876dbca3e471f, 0x951937d8682670ce, 0x0f99cb1f52ed466a, 0x8c7cd205543b804c,
+     0x2fd24d74a9c33783, 0xe5dcb7b7762e5af1, 0x45e6749cca4af77c, 0x540ac7ee61f2259f,
+     0x89c505c72802ce86, 0xeab83b9d2d8000d1, 0x9f01d5e76748d005, 0xc740aaef3035b6d0,
+     0x49afcd31d582d054, 0xcba5dc4c1efb5ddc, 0xc0a4c07434350ca1, 0xfc8dfaddcc65ee80,
+     0x157c9780f6e4b2d9, 0x9762a872e1797617, 0xc4afae2cf3c7e1bd, 0x71cde14591b595d4,
+     0x8843c3e0e641f3b9, 0xd92ecd91dce28750, 0x1474e7a1742cb19f, 0xec198e22764fa06b,
+     0x39394edb47330c7d, 0x00ba1d925242533d, 0xaed8702536c6fb30, 0x6d3618e531c2967a,
+     0x77f7cedcd7cc0411, 0xbc1e2ab82be5b752, 0x07b0cf9223676977, 0x596c693b099edd53,
+     0xbb7f570f5b9b2811, 0x96bfdad3c4a6840c, 0x668015e79b60c534, 0x3ad38d72123f1366,
+     0x6b994d81d2fcbb09, 0x70885f022c5052d8, 0xc891ee79d9306a7b, 0x2c4df05c0ed02497,
+     0x19ebc13816898be2, 0xea7c64df11c392a2, 0xb7663e88dd12e1bd, 0x79f768cb8e154c21,
+     0x1fb21b12e945933b, 0xe6a9045643f6906e, 0x544c47acd7e15371, 0xb7709b14f727e3d1,
+     0x326ee36a46942971, 0x477f1cf7b0e2d847, 0x88b8f6b82b3b0c24, 0x18bc357b80e3cd5c,
+     0x3333de70e4d66e0b, 0x4fd4c5e148583cf6, 0xae1b62f3008c0af3, 0xc49f419b6ab29cf5,
+     0x2c29fa65afc3fa28, 0x4b19d93734d03009, 0x7dd6c09e589276ad, 0x1cece97f30de48ad},
+    {// seed = 4
+     0x58bdf4338602e4fb, 0x71a5620b02c926d5, 0x3811c960129c2d9f, 0x29c2fb11fccac567,
+     0x0d6b1ea7780f1352, 0xcc4d3ddfae3f87b3, 0xfdd30257362a586b, 0xabc948fde69f25f1,
+     0x51b3523469d30f7b, 0xe0f0322724405ace, 0xd3729266d896da1e, 0xb10c37e5147915bf,
+     0x8b577039f9fa32a3, 0xe677c6a9cbfb44b3, 0x7317a756ebb51a03, 0xf8e988ef37359485,
+     0x600fc1ef3f469ff3, 0xbf0b8f8520444e01, 0x3711168b08b63d73, 0x34146f2944a6cb36,
+     0x717feb263862cdde, 0x7185f8347db00412, 0x900798d82127e693, 0x84089e976a473268,
+     0x10f8308c0d293719, 0xf62a618d4e5719b8, 0x8bdbd257a1a9516f, 0xf49f666fd7a75110,
+     0xbaf45e2db7864339, 0xe4efa1ea0c627697, 0x3e71d4c82a09fe10, 0x54a2a51cf12127bb,
+     0xa0592c9f54ba14cd, 0x27dd627a101c7a42, 0x3d2ceb44b3d20d72, 0x7ee1f94a68ca8f5d,
+     0x7e8cb8651b006c36, 0xbd9fa7ca3a475259, 0x856de173586a7b34, 0xcedb291b594cb1b5,
+     0xa3d6e462fd21cddc, 0x74561d10af9118e4, 0x13a3d389fc2d4b36, 0xeea8594a4a054856,
+     0xf56d7474d9ba4b13, 0x25ddce2f6490b2fd, 0x920653ff3a8d830b, 0xcd8c0c9cdac740d1,
+     0x2c348a738db9c4a0, 0x2967ccbe8ea44c22, 0x47963f69adb049f8, 0xf9d01eb5b4cf7eb6,
+     0x7a5c26eb63a86bd2, 0x62ad8b7a71fa0566, 0xb373213179f250ae, 0x589d4e9a88245a4d,
+     0x433dafebe2d558a8, 0x521fbef2c8fe4399, 0x62a31f9ff9ccd46b, 0x51602203eba7c1a6,
+     0x9afc8c451b06c99f, 0xb529085bdbaffcea, 0xac251825cc75892b, 0x94976a5bce23d58e,
+     0xdd17925b6c71b515, 0x568fd07a57bce92e, 0xefac31200d8bd340, 0x716c3e466b540ef9,
+     0x3d2c9e380063c69b, 0x14168f9a3662dd83, 0xd298c7504dbc412f, 0x74490a94f016719f,
+     0x0e0da431e1ab80c8, 0xe321f63dc6b169ae, 0xf08671544febc95a, 0x39324450cc394b3b,
+     0xea6e3d35f1aa3a70, 0x8ef8a886508ce486, 0xdc1a631ef0a17f06, 0xfda2b3fbcd79e87b,
+     0xd75bcae936403b10, 0xf88b5bd9f035f875, 0xc43efec2e3792dd4, 0xe9fac21a9d47cd94,
+     0xc2876f0c4b7d47c3, 0xaba156cf49f368b4, 0x5ccda2170fa58bf9, 0xadc92c879ed18df7,
+     0x110c1b227354e6c8, 0x298ee7a603249200, 0xde92142ede0e8ee7, 0x88e4a4610644ba9e,
+     0xbb62d277e7641d3a, 0xb9be1985b7bf8073, 0x29024e5426cdb0d1, 0xf6aefd01f3092ab8,
+     0x2a07087b313133aa, 0x6d71f445d6dfc839, 0x1e2412ff12e5526b, 0xed5cdeba6617b9e1,
+     0x20b1d0d5e5f8760e, 0x12ff15705c368260, 0x7bf4338b7c387203, 0x34ff25f00cd06185,
+     0x1148c706c518cf28, 0x5c04f0623388f025, 0xcb9d649275d87d79, 0x9b5f0c24fabc42ec,
+     0x1a7b5e7964e33858, 0x2a81bbd8efdc6793, 0x8d05431ffe42752e, 0x83915cd511002677,
+     0x580ed4d791837b31, 0x5982e041d19ff306, 0xcad0d08fa5d864ca, 0x867bee6efe1afa63,
+     0x26467b0320f23009, 0xd842414dfda4ec36, 0x047fcdcbc0a76725, 0xbddb340a3768aeca,
+     0xef4ce6fa6e99ab45, 0x88c5b66c7762bf9b, 0x5679f1c51ffb225d, 0xdab79048317d77ee,
+     0xf14e9b8a8ba03803, 0xe77f07f7731184c1, 0x4c2aab9a108c1ef5, 0xa137795718e6ad97,
+     0x8d6c7cc73350b88b, 0x5c34e2ae74131a49, 0xd4828f579570a056, 0xb7862594da5336fc,
+     0x6fd590a4a2bed7a5, 0x138d327de35e0ec1, 0xe8290eb33d585b0b, 0xcee01d52cdf88833,
+     0x165c7c76484f160e, 0x7232653da72fc7f6, 0x66600f13445ca481, 0x6bbdf0a01f7b127d,
+     0xd7b71d6a1992c73b, 0xcf259d37ae3fda4a, 0xf570c70d05895acf, 0x1e01e6a3e8f60155,
+     0x2dacbb83c2bd3671, 0x9c291f5a5bca81af, 0xd976826c68b4ee90, 0x95112eec1f6310a2,
+     0x11ebc7f623bc4c9a, 0x18471781b1122b30, 0x48f7c65414b00187, 0x6834b03efa2f5c30,
+     0x0875ef5c2c56b164, 0x45248d4f2a60ba71, 0x5a7d466e7f7ba830, 0x2bebe6a5e42c4a1d,
+     0xd871d8483db51d10, 0x6ee37decd2fd392f, 0x7d724392010cede3, 0x8e96ef11e1c9bcc8,
+     0x804a61d86b89d178, 0xbb1b83ce956055ec, 0xcb44e107410ff64f, 0xc426bb09ee0ba955,
+     0x057c08f42c3dd7f1, 0x40ea1ec148602bdf, 0xc24688deeb65d7f1, 0xd8bcc53c768ba4e4,
+     0x16e0e3af65c1106c, 0xfc12f7e7d647218b, 0x70d6e1d3ee93cef4, 0x01d2a505c4541ef9,
+     0x1ef79e16e764d5c3, 0x0363d14d13870b98, 0xb56ef64345d06b11, 0xe653d557ebb7c346,
+     0x8304a8597c2b2706, 0x1536e1322ce7e7bb, 0x525aec08a65af822, 0x91f66d6e98d28e43,
+     0xe65af12c0b5c0274, 0xdf6ae56b7d5ea4c2, 0x5cef621cedf3c81c, 0x41e8b1ffd4889944,
+     0xb5c0f452c213c3e5, 0x77af86f3e67e499b, 0xe20e76ea5b010704, 0xbdc205ab0c889ec0,
+     0xc76d93eb0469cd83, 0x17ac27f65cab0034, 0xd49ec4531fd62133, 0x07a873ea2f1b9984,
+     0xbff270dfef0032ee, 0x1764dbe91592f255, 0xe40363126f79e859, 0xa06cad3ab46971f6,
+     0x0be596e90dedd875, 0x3387cce5c1658461, 0x44246acf88a9585e, 0xe0ad82b92d5ecb2c,
+     0x2177491c9a1600a6, 0x16e7c4aac0f02422, 0x75792eeeec15c4e1, 0x2309cd359d08ee30,
+     0x7cd9831dd1b83b0a, 0x374914a7c4ee8cf0, 0x0dd17765c9ac2e54, 0xb7847470ba9a7688,
+     0xfba4f4bbe2991173, 0x422b203fc3de040e, 0x63bfcaf2ecf2ab0e, 0x0c5559f3a192946e,
+     0xfdf80675c1847695, 0xf5f570accab842c9, 0x65cc5a448767afea, 0x1efeb0a7ee234f2f,
+     0x9b05f03d81e7b5d2, 0xe7c31317a8626cf4, 0x620f2a53081d0398, 0x1b6de96cdd9943ae,
+     0x8c226a436777d303, 0xa08fbbd50fafb10d, 0x6a64c5ec20104883, 0x9c9c653502c0f671,
+     0x678a02b2174f52a0, 0x68e008ba16bbad4b, 0xa317c16d2efb860f, 0xeab2075d17ed714c,
+     0x565eeeddf0c4ea15, 0x8ec8e94d242a6c19, 0x139e8e27d9000fae, 0xc977a7ff1b33d2f5,
+     0x1d0accca84420346, 0xc9e82602cd436e03, 0x6a2231da53d2ccd3, 0xb44b12d917826e2a,
+     0x4f4567c6a74cf0b9, 0xd8e115a42fc6da8f, 0xb6bbe79d95742a74, 0x5686c647f1707dab,
+     0xa70d58eb6c008fc5, 0xaaedc2dbe4418026, 0x6661e2267bdcfd3d, 0x4882a6eda7706f9e,
+     0xf6c2d2c912dafdd0, 0x2f2298c142fd61f9, 0x31d75afeb17143a8, 0x1f9b96580a2a982f,
+     0xa6cd3e5604a8ad49, 0x0dae2a80aad17419, 0xdb9a9d12868124ac, 0x66b6109f80877fac,
+     0x9a81d9c703a94029, 0xbd3b381b1e03c647, 0xe88bc07b70f31083, 0x4e17878356a55822},
+    {// seed = 5
+     0xb3c58c2483ad5ead, 0x6570847428cdcf6c, 0x2b38adbf813ac866, 0x8cb9945d37eb9ad3,
+     0xf5b409ec3d1aed1c, 0xa35f4bffc9bb5a93, 0x5db89cde3c9e9340, 0xff1225231b2afb2b,
+     0x157b0b212b9cc47d, 0xf03faf97a2b2e04d, 0x86fdab8544a20f87, 0xfcb8732744ae5c1c,
+     0xd91744c0787986d5, 0x5f8db2a76d65ad05, 0xcff605cbed17a90d, 0xf80284980a3164e7,
+     0x59cc24e713fccc7d, 0x268982cada117ce4, 0xcd020e63896e730e, 0xe760dc46e9fe9885,
+     0x6aaece8ab49c6b5d, 0x7451194d597aae3e, 0x35d4385900332457, 0xa40fb563a096583d,
+     0xa797b612f7f11b76, 0x2fed6eb68e6a2b9b, 0x2f06ee64aeffd943, 0x9dd0e49d9ca45330,
+     0x97d48f08bd7f1d8f, 0x1cfa7fe3ebe4d8ee, 0x2a2ba076bd397d42, 0x68c4344f7472f333,
+     0xce21ec31987d74b5, 0xb73dabdc91d84088, 0x801aadee592222fe, 0xaf41345398ebc3f5,
+     0x8a8f653d7f15ee46, 0xce2d065ff2ba2965, 0x4e05da515da2adb7, 0xa6dbdb8aa25f0fd4,
+     0xca9f9666bbd2d5a9, 0x6b917ce50bd46408, 0x1550cc564ba6c84d, 0xb3063ae043506504,
+     0x84e5f96bb796653d, 0xe2364798096cf6e3, 0x3b0dfedf6d3a53d0, 0xb7e4c7c77bde8d93,
+     0xe99545bac9ab418a, 0xa0e31f96889507bb, 0x883c74f80c346885, 0xf674ae0b039fd341,
+     0x8bb6ce2d5e8d1c75, 0x0c48737966a7ed7c, 0x04fcdf897b34c61c, 0xe96ac181bacbd4d6,
+     0x5a9c55a6106a9c01, 0x2520f020de4f45d3, 0x935730955e94d208, 0xce5ad4d7f3f67d3b,
+     0xa4b6d107fe2d81ca, 0x4f0033f50ae7944e, 0x32c5d28dd8a645a7, 0x57ce018223ef1039,
+     0x2cbab15a661ab68e, 0x6de08798c0b5bec2, 0xee197fb2c5c007c6, 0x31b630ac63e7bda2,
+     0xab98785aefe9efe3, 0xa36006158a606bf7, 0x7b20376b9f4af635, 0xa40762fdc3c08680,
+     0x943b5faffd0ebee2, 0x7f39f41d0b81f06e, 0x7c4b399b116a90f8, 0x24e1662ac92bc9f3,
+     0xcf586fc4e8e6c7db, 0xe46e0d047eeb12d7, 0xe8021076e4ea9958, 0x11fc13492e3ca22a,
+     0xd61eae01410397e3, 0x7e8c4a58036a8e9f, 0x068a6de267970745, 0x64faab129bef1a41,
+     0xb4a6f720943dad01, 0x631491058d73a9d5, 0xdad4fe95eab3ec02, 0x0a8b141c5c3a44f6,
+     0x9fc69d4c2b335b98, 0x94d5f84a07d6e4cd, 0x1b73965de143c608, 0x443932c2dda54bcc,
+     0x7397818fb0b04cd2, 0xef4ab03a1202b277, 0xf3d2ee459c0c2b92, 0x182d4daf8b058a87,
+     0x90e63035d7b51368, 0xba4cd8b9a95d45fd, 0x12a7392c76731090, 0x890d264ec5d082d2,
+     0xeeaf5c363da4994e, 0xd6aad756902123fb, 0xb531ebebdb28f191, 0xe71ce659fc59babd,
+     0x37c1b94f63f2dcb5, 0xe4e3abeb311f9b96, 0x4a31b72ccb8695d3, 0x52cae1f0629fdce4,
+     0xe5b0475e2ed71369, 0x2724e8c3506414fb, 0xbab0367920672deb, 0x0161a781c305449f,
+     0x37b70f40f5bb60be, 0xddd1094c50251a01, 0x3b28283afd17224e, 0x06dec0cfe889fc6b,
+     0x47608ea95bb4902d, 0xad883ebc12c00e82, 0x9e8d7ae0f7a8df29, 0xa79443e9f7c013a1,
+     0xcfa26f68b7c68b71, 0x33ae6cc19bda1f23, 0xd9741e22b407887f, 0xf2bff78066d46b1c,
+     0x794123191c9d32d4, 0x56cb6b903764ec76, 0x98775d0ef91e1a5a, 0xae7b713bc15c1db9,
+     0x3b4c1a7870ed7a0d, 0x46666965f305cc34, 0x0ea0c3b2e9c6b3cd, 0x4dc387039a143bff,
+     0x5f38bb9229ef9477, 0xea5d39ba72af7850, 0x69a5ed0174ce2b6d, 0x06969a36bfe7594d,
+     0x0adee8e4065ccaa3, 0x908a581d57113718, 0x64822d6c5a8190ed, 0x8c5068b56ace4e4c,
+     0x88ba3b4fb4e30bef, 0xa6ec0b8bb5896cfe, 0x4e23fcc6b47996fd, 0xe18e75b0dd549c7a,
+     0xcd90f17e106cf939, 0x1666fdfb2ef7c52f, 0x4fae325f206dd88c, 0xe7bc1160e25b062d,
+     0x3cc999cb246db950, 0xc5930a7326cd5c37, 0xb008a48a211367bd, 0xc5559da145a88fd4,
+     0x1e3ad46655fac69c, 0x7834266b4841bfd7, 0xa764450fbffc58cc, 0x54d8cf93a939c667,
+     0x93c51f11b21b2d9d, 0x0964112082ed65cc, 0x4c2df21213e7fb03, 0xf0405bc877468615,
+     0x17b4fc835d116ab4, 0xa6b112ae5f3cb4ef, 0x23cfc8a7fd38a46e, 0x8e0a360dc2774808,
+     0x24ca9c8092105ad5, 0xafd3f75524f2e0d5, 0x4f39ed7dbaddc24c, 0xe5e362c7679a7875,
+     0x00914a916b07b389, 0xdfe1119b7d5ab5da, 0xabd6ed9940e46161, 0x630ed2044171e22c,
+     0xdecc244157dd1601, 0x777e6d5b4b4868d5, 0x9b3530bee67017d8, 0xd2faf08b291fdcb9,
+     0x006e99455d6523de, 0xd559b5817f6955b5, 0xefcc1063b0088c61, 0xed73145ae0f00ae7,
+     0xab2af402cf5b7421, 0x897767f537644926, 0x26c9c0473ca83695, 0x192e34e1881b2962,
+     0xf7cf666ec3b3d020, 0x27f9b79c7404afb7, 0xe533e8bed3010767, 0xe5817838e11d05d3,
+     0x65659c531bd36517, 0xd427c5e0a23836fd, 0xf3eab7ea58fa3528, 0x07683adae1289f35,
+     0x201d6af7e896dd32, 0xd5da938b9a21ad88, 0x843fb73ad67bc316, 0x1782ec7d5feef21b,
+     0x943f66f6ec772877, 0x7e9112e7b26da097, 0xeac8161f8663c2c7, 0xe8600db480a9ebf4,
+     0x07807fc90f6eaf5f, 0xe0e4c9deb41abf83, 0xbdf533db271f9c15, 0xb398411b0497afe2,
+     0xdebb45ef25448940, 0xe7a5decefcd376c4, 0xaf1ef3c728c83735, 0xb8b83a99355cb15a,
+     0x6444a0344f1611e4, 0xe8bb7f5cf3c60179, 0x77ab5c5177e75ff7, 0xc38fd6fa849d585d,
+     0x390d57d53029060a, 0xa66327eb7b8b593c, 0x6350a14f6fcd5ac9, 0x2c08125bcd7008b4,
+     0x2d00c299a6a6bf8e, 0x6b0039c1f68d1445, 0x0035150c5d06f143, 0xa34d01628cc927e1,
+     0xdf5b3164d7b2ede1, 0x8167db1d0583d72e, 0x4e13b341cd2ae8bc, 0xa693d9b1f416e306,
+     0xc15ed7ca0bc67609, 0xdc344313c1c4f0af, 0x88b6887ccf772bb4, 0x6326d8f93ca0b20e,
+     0x6964fad667dc2f11, 0xe9783dd38fc6d515, 0x359ed258fa022718, 0x27ac934d1f7fd60a,
+     0xd68130437294dbcc, 0xaf5f869921f8f416, 0x2b8f149b4ab4bf9f, 0xc41caca607e421cb,
+     0x7746976904238ef9, 0x604cb5529b1532f0, 0x1c94cd17c4c4e4ab, 0xe833274b734d6bbe,
+     0xe9f1d3ef674539ce, 0x64f56ed68d193c6a, 0xe34192343d8ecfc1, 0xcb162f6c3aa71fe8,
+     0x99eaf25f4c0f8fa4, 0x92f11e7361cb8d02, 0xb89170cddff37197, 0x4f86e68a51e071e3,
+     0x31abf6afd911a75b, 0x6d20cf259c269333, 0x4150b9f88fcb6513, 0x705063989ebf7451,
+     0x559231d927c84410, 0x1ca8ec4b098bc687, 0xebed22405c9180e0, 0xaa815b37d052af59},
+    {// seed = 6
+     0x946ac62246e04460, 0x9cebee264fcbc1ae, 0x8af54943a415652b, 0x2b327ed3b17b8682,
+     0x983fde47b3c3847e, 0x10a3013f99a2ad33, 0x6e230bb92d2721ef, 0x1cf8b8369e5c5c50,
+     0x7f64017f2b7b3738, 0xd393248a62417fa1, 0x9ff01c0b20a372c5, 0xb0e44abce7e7c220,
+     0xcebb9f88d48a815f, 0xdb7df6bd09033886, 0x7844fc82b6fa9091, 0x72d095449863b8ec,
+     0xc13e678c89da2c7e, 0x6caf4d5ad231d12f, 0x2e0ab7b5fcf35c49, 0xf410720cb932a70f,
+     0xd66ea581f16fce06, 0x175c9f002f57dc98, 0xccbcfd0d32988775, 0xfde4c407d3b0a232,
+     0x5db2931ae7e97223, 0x6e07e2173085809f, 0x6e1d1ec0f9cad73c, 0xb2fc251a7f802619,
+     0xbc1fc17f04f342de, 0x8de8f21ec658e078, 0x72c0f40cbee53fd6, 0x0678244411fc17a1,
+     0x1d5837ca166b9bbd, 0xc8cada003c554345, 0x6a2fe2bfb2e58652, 0xfca9d797a6f7988b,
+     0x6699e24ac737948b, 0x69623ffcb05789ba, 0x946429c529d95b75, 0x0d14df0b2a13970f,
+     0x593d8592c440dfec, 0x2ee176f3d7e74b94, 0xae003f1da3be9e26, 0x0c7b02c4c0f6764a,
+     0x3117e2fa1f632462, 0xf0f23265b6f1eaeb, 0x3111255d9b10c137, 0xc82745e509a00397,
+     0xbd1d04037005fea7, 0xe104ab0dd22a9036, 0x51b27ce50851ac7a, 0xb2cb9fb21b471b15,
+     0x29d298074c5a3e26, 0x6ebdf2058b737418, 0xc4a974041431b96f, 0x1ec5a30ccb6bdaac,
+     0xe818beede9bf4425, 0x4b69b1bce67a5555, 0xf5c35f1eb0d62698, 0xf4509bbd8e99867c,
+     0xb17206debd52e1bc, 0x35785668c770b3be, 0xe9343987ff5863bc, 0x2ee768499ac73114,
+     0x5132bb3426eeaaf4, 0x471bce2c6833c5ff, 0xbb9a2d5428e6f6f9, 0xd5678943c595792d,
+     0xab2a65e7f81e479c, 0xa82407bb23990b31, 0xdae321383984923c, 0x01823bb22648e6f1,
+     0xda6e8df4214a8b04, 0x0e172bb88e03d94f, 0x552da6c22e362777, 0x7ce67329fb0e90cb,
+     0x7b2d7f287ede7ebf, 0xd44f8222500651bd, 0x4acca1ef58fbb8ab, 0x428ecf058df9656b,
+     0xd7e1ec6a8987c185, 0x365be6a54b253246, 0x168849be1e271ee8, 0x6a00f3c4151a8db2,
+     0x37602727ca94b33d, 0xf6b50f18504fa9ce, 0x1c10817f6bc872de, 0x4bfe1fe42b0f3638,
+     0x135fad4b8ef6143b, 0x1b25ad2bafc25f58, 0x41e37f85cf321f92, 0xfc73f75d9d5b9bea,
+     0x9eb3694d1e9cb7e1, 0x601d51f08fa83b90, 0x234a2a9b88366f41, 0x63fe903e16f2c3bf,
+     0x1cdbd34fa751c0b0, 0x0ce4fc6747c0558c, 0x51ed72afb8bb49aa, 0x20313ba13ca12c96,
+     0x271fa38f9ebd54c1, 0x3696a5ac03a8edde, 0x05602be7df625702, 0x11f1ac73790f7a9f,
+     0xa2836c099f0810bd, 0xe5ac2e47caa532fa, 0xd9c000a66d39f681, 0xd93d900e6f3d9d5f,
+     0x792c81c65b7900f2, 0x5c5dce790ee20da1, 0x74ff1950edec1aee, 0x71fc85fa1e277d8f,
+     0x0e77df17d6546cbc, 0x07debad44816c3b4, 0xbafa721581e92a70, 0x8ab6fbe2ed27bba8,
+     0xe83243a20dea304a, 0xaa85a63a84c00a07, 0xde0e79917fc4153a, 0x21bb445e83537896,
+     0xeedcac49fc0b433a, 0xffb2926a810ae57a, 0xf724be1f41d28702, 0x79cb95746039bb3b,
+     0x5a54fe3742a00900, 0xda4768d64922c04f, 0x420396a84a339dae, 0xa171e26ee5e8724e,
+     0x4c8da7c5d289c20a, 0x9ebd79a1a8e94742, 0x39235232b97e9782, 0xb75df0be9bba7d80,
+     0x0c1d204dd87d48fc, 0x8f81f3e7177266e8, 0xe4a460b39e78d72b, 0x50b98fa151e65351,
+     0xb7cb585c3ee1eddc, 0x11cdad9a76ee1dc4, 0xa38054a78595dc1c, 0x92f09e2ec4978edc,
+     0xa8f0061b5efdabaa, 0x04bcc4abc224d230, 0xc58606738e692d46, 0xdd2b27b565952433,
+     0x19e6ed1b740beec0, 0xceadd49b2ef9891f, 0x328178c28fe95cad, 0xe5ad4c43afe02848,
+     0x03c0cb538cd967c0, 0xec4352526d19a630, 0x4c7e99389d39b031, 0xf65dd05362c2deb6,
+     0xd1e70daf6879d28d, 0xbe9f57db6309b265, 0xa4b66f370b872bb7, 0xe26896fbc6ee1fd5,
+     0xac705e661bfcf7c5, 0xab4d0d07d7f09940, 0x976417c06aeb6267, 0x8161c684a6bd468c,
+     0xf77b6b9976dc4601, 0xc6489b779a39c12c, 0xb2aa58d5681cea1a, 0x043b1b40f8c3e04c,
+     0x681fcbfadc845430, 0xab8896c921ba8def, 0x57aaf172606f37b2, 0xc3735048cd5eb8d7,
+     0xa7078b96955631bd, 0xdd6b3543aa187f33, 0xc7103ea4a2a697fd, 0x8d7b95f6ff1f7407,
+     0xe44f419e84709530, 0xf340caa9132cbb0a, 0x2ba407283143c66c, 0xe1be240ca636c844,
+     0x90d32f2877ac08bc, 0x5d26e6294b2c8673, 0x4a6b2f5b27c87a44, 0x961fb9043f76d34f,
+     0x0afee02d8d3c55d2, 0x6228e3f48c42e5dc, 0xc338e69ee6593675, 0x853f74b16efb7bdd,
+     0xd062f40bdd22e687, 0x647164b9ab4c4190, 0xf94689f67d598369, 0x8e4b29d87a5012d7,
+     0xaf02b8b925656fbd, 0x7a722a767179a630, 0xb5c8afe937a75ace, 0xfdb8e8d02d279372,
+     0x887ef700cb25fae1, 0xcfe9bd912f72cabe, 0xb1d4dedc24f978de, 0x517522d38319cc2a,
+     0x7dd87b2b36aab798, 0x579c4ff3046b5a04, 0xf5c5975c5028b7a7, 0x7094579d1000ec84,
+     0xbc8d5b1ea70a5291, 0x161b2d783be8855c, 0xd26d0b0d6d18279f, 0x0be1945f02a78bd5,
+     0xb822a5a9e045415b, 0x2fe9d68b1ccc3562, 0xb2e375960033d14f, 0x26aca04e49b4ff22,
+     0x732a81c862112aea, 0x8bd901ed6e4260b8, 0xe839532c561ad5b0, 0x8fb6e4d517a79b12,
+     0x0dd37f8c0be9b429, 0xc8ad87ad12f1b1b0, 0xc51f3aa62b90318b, 0x031a7e8b86c1cefc,
+     0xa95547af2b70fc76, 0x9cb3615c5a98801e, 0xa387e3c3341d7032, 0xa087ea52a1debaef,
+     0x16325ec9a2e6e835, 0x587944a484c585eb, 0xc8879033bde22ecc, 0xa39dbfce709c464a,
+     0x7acc010f99208774, 0x98dd2973a096c5ad, 0x26458b51139f198c, 0x2f5d19575e8c4f02,
+     0x726643f0d38af352, 0x44d879b6d73e6e94, 0xa68a03885c980abe, 0x06048acd161c40c0,
+     0xa4dab8f89d405d28, 0x7120c880cb04be18, 0xa062ace22a1cf0cf, 0x3901a9daf29704f4,
+     0xff08f3ed989db30a, 0x6d22b13e874c67e9, 0x80c6f35518d73f4d, 0xc23c2a521aac6f29,
+     0x2e708fd83aaa42e0, 0x7fc3780f55f1b0fd, 0xabb3075c98cf87f2, 0xb4df3f40f7c61143,
+     0x2a04418098a76d75, 0x0d9eeee9509b2d37, 0x6be8ae51f4b59cdc, 0xe746cc7c00e4a2ab,
+     0x785bc6df9cac597c, 0x33cb6620ce8adc48, 0xc1ba30739bffcef7, 0x6d95771f18e503f7,
+     0xf7be3ae2e62652ff, 0xc8d82ffd2a73c62b, 0x8725a3ba5b110973, 0x67ed6b9c724757ec},
+    {// seed = 7
+     0xc0272d42c19ff3ae, 0x4694228b43ea043b, 0x5709a6ef8a462841, 0xc9210a1e538805c9,
+     0x279b171196113ec2, 0x859b769fc2d9e815, 0x0d5d3125a2bf14d3, 0x22bca1cfefa878ba,
+     0x481b6bf58037bd83, 0x4933ba8647728d22, 0xf08c7b6b56f6e1b6, 0x374e8af5a15407c7,
+     0xa95c4dc3d2487a5c, 0x9b832808ff11e751, 0xf2048507e9da01d5, 0xa9c576189f544a4a,
+     0xf6c2a45b2e9d2b41, 0x9b9874c9f10ecc2f, 0x37d9b5f51f8c149e, 0x93aead54c9de9467,
+     0x59cf0b4af262da23, 0xe7e9929af18194b2, 0x9df2644e33eb0178, 0xde4122d6f0671938,
+     0xf005786c07f4800b, 0xb1fc9d254b5d1039, 0x0bf1088631f6dd7b, 0x665623f0a4b8f0c7,
+     0x60f0113a9187db7c, 0xfd7cceda4f0d23a6, 0x26c01e9d89955940, 0x33afa1dfc0f5a6a0,
+     0xeb77daf215e9283c, 0xc7575214bf85edb4, 0xeb0d804bf297e616, 0x84bff4ffd564f747,
+     0xc4ac33189246f620, 0x43ef61213ecc1005, 0xcbbb0dea6cd96acd, 0x8ed27abfa8cfcb05,
+     0x543b61529cb996b6, 0xa5f987ca41ea5e59, 0x3c50e0ac5254cb7a, 0x4192b0446c06d1e6,
+     0x3e86592e21b45388, 0xdb766f06fcc6e51e, 0x0448ee36efe632db, 0x663c9db689253e35,
+     0x72e0bd4985331dd4, 0xff501b5bf7d94e74, 0xe911ce758e2113a8, 0xec3a8d03a75a6ba4,
+     0xaf6b4b72f56edc83, 0xf284857936c0a391, 0x5ba6feff407d46f4, 0x9d689c26de9d6702,
+     0x28c04a9083726b5d, 0x2ccf4a627a029730, 0x7b4719500d4f0c71, 0x76470a9a7da250a8,
+     0xcc48409404a1c890, 0xccefbdc7ec9a8055, 0xe0db91bff3cc42d3, 0x0532436426141254,
+     0xf2ee9325e6f0ff0b, 0x149c20a5fbb28d9d, 0xe71624cd8d2d14d4, 0x8f01d4dc8cc2dd77,
+     0x29cf409b333015b7, 0xba8bebd211884dd1, 0xc3396635e8c8db1d, 0x8ed0f6208d0528b8,
+     0x0d90b43fdd0ee334, 0xd73c9a3333a044c7, 0xa2595cd208dbdc38, 0xae93cb264f940c09,
+     0x8e0538d8afb07a97, 0x19115ec881385ba2, 0xa886f9e6a8039c6a, 0xcd5d62147ce3ecac,
+     0xaecdf9e0bb4969f7, 0x2ddd631c53dcad10, 0x73ad1c97b3412054, 0xb08915fa2722efc6,
+     0x97966047e5067eb0, 0x337f1675ed91445c, 0xb3a833d150b96a0d, 0x5940a98fe35e5e2e,
+     0xfd03cc354ed0d8ff, 0x4e65b98291a8644a, 0x14a259f2852a60b2, 0x7648e3478c1e8e5f,
+     0xbc0fbef6d9a919b4, 0xbec4302081346cf1, 0x57d2ce7aa1c7c511, 0x234c209d8f4e1ac3,
+     0x87cf80cc933ce443, 0x7c262c616931e94e, 0xc5e33b049cf9eddf, 0x1a80790ed03ae51b,
+     0xf2e8b9494f7220cf, 0x124cb59c14fff3ff, 0xa8a06cbfdb86ce18, 0x9068ef1f80b37653,
+     0x0c55417b8d90338f, 0xcd579a523f6bcd30, 0xa31bfe2476a8d2a9, 0x1f8d142208094223,
+     0x332dc40a5203cfad, 0xf8792fe5b2d33b4c, 0x443bd9668bf9461e, 0xc9019db0ace1409e,
+     0x781bea919a113e8b, 0xb0f11d866abfbeec, 0xcfe139a60db0c26a, 0x869ab8721e6aa39e,
+     0xdb48a4977717837a, 0x588a5ff151065b18, 0xe4a251ea0028864d, 0x7f0e43ba408a77c3,
+     0x65f66dd50a536135, 0x6f49e934d9331c3e, 0xb8d742e0f0fa6b09, 0xe4e9b272deca2348,
+     0xaee132ff902f773c, 0x43f658f7c2a0c90a, 0x28cb4dbc76cc53ea, 0x7d92253aa99ac39b,
+     0x4fea3d832370baab, 0xb29e36936e51d78e, 0xea10778712321064, 0xff4f21f8ef274be2,
+     0x84eff18ddfa0933f, 0xd0ec6a9f86c758a0, 0xaf82e5973c431ae0, 0x352023c00c045425,
+     0xad34d7bc4a2f8961, 0xbdb4a02a24d4dee0, 0x354a4846d97447cf, 0x331a8b944d5bc19f,
+     0x5ce04f8e17909035, 0x6497581bad8f4aab, 0x07c503bba647111e, 0x85f412ba78e1f7ff,
+     0x7f3b920fd20f4cff, 0x424e1a9a4ce34e2f, 0x3035e2d62e1b9f0a, 0xef63114bff7b729a,
+     0xe86a05889ab6bb60, 0xee0830cf095585a1, 0x4a54f7fa47d9c94b, 0x17daeece9fcb556a,
+     0xc506d3f391834c6f, 0xb3f24be362e1af64, 0xc435e4e23608efdd, 0xeeba9caaa4cc1768,
+     0x5a71f306daddc22d, 0x18e5205f41eba1a0, 0x7b29b4d1f6610925, 0x065cb65a0258d9a9,
+     0x3e5ac8faa9fd1f95, 0x3b362362c1ea0470, 0xce0e4f6434db7a2e, 0xf327341098de52f2,
+     0xcfca3b9e2a1992c3, 0x7483bf9401233e41, 0xbafbac531c6f9281, 0x4b52dd71b2c106f8,
+     0xdf73b66e50b5a1f7, 0x237aec0202a20283, 0x23dd5be23dffdf2b, 0xea9730731ee122ef,
+     0x5cb3f846014fbcd3, 0xc3b21c8ffdce9201, 0x06a99a02f91a8760, 0x721a81fa8fd7b7a3,
+     0x6aafcdddc53cbcd8, 0xd03b464005a93bcc, 0x8212edc1b1669dcb, 0x71f4c31364c31bc7,
+     0xfeeec0eba8772307, 0x1948d00a13d88cf1, 0x19064fd6d943ada8, 0x4ec8d31722697bfd,
+     0x596d9a953a516609, 0xc4cb4bff53507da2, 0x1d59f3c5be36e4ca, 0xe5b4fc5bf6044c9b,
+     0x1bb74e052232f735, 0x04e8a0db611ddd5d, 0x8d04eaa009b421bf, 0xa7878ae0ac0e6d58,
+     0x28c1030217cab2b3, 0x827943767e56a883, 0x28fce5fa02d22809, 0xb30c322fffc8c58e,
+     0x1ca5a6a9f8066c5b, 0xb24db5f1462b2513, 0x02f653b89b7e5f6c, 0xe31f8fb5d5f78eee,
+     0x266acc514ed93501, 0x936879d1c6fddcc4, 0xcd51be3636af1952, 0x3fdbb6fc332c78c8,
+     0x9eb656379fa73094, 0x056146cc92fa0f96, 0xed6c4f1836c027c3, 0x021e0bb5d2113f2a,
+     0x8983e42ec1c626b3, 0x73ea9bc6513ad9c9, 0x0c904903b24f4247, 0xacbac1e6243e2525,
+     0x0b1069a0c230fb06, 0x77d709fca3fc1ce5, 0x87ad0f65020947e6, 0x555302641c53f4e6,
+     0x65ea87871fa9aaee, 0x58aaf4ecc1067bb4, 0x1a66c48cc4c65b3f, 0xca96aca48b2ea969,
+     0xa68eb70bad14de2b, 0x5ccdb3d7e00a6f6e, 0xe178fbfec73fe72f, 0x2b63d6a16b83e890,
+     0x32fdb7a5330fbae0, 0x2ab5803c8d1bf32c, 0xda838388c1527c94, 0x16a50bdc4de24acb,
+     0xe561301f134c074a, 0xd7ae63d2816b4db1, 0x036aabd4df0dd741, 0xc5e0db8783435b9d,
+     0x9c4386cf0a07f3b2, 0x6a72ac1aa56a13a1, 0x299bbdb04bb20a23, 0x138c1018fda16b81,
+     0x0e354f0b3bda49df, 0x9f4c295b23127437, 0xd133ceb2bd561341, 0xd8b4bfd5a526ac29,
+     0xcdd0a70ddc1c7bbd, 0x81dce595bf572225, 0x1c6f925c05f6efd7, 0x8ae5097553856ea0,
+     0x3aabeaeef248f60d, 0xd9005809d19a69e2, 0x2a3a1a314311cc27, 0x89bb2dc76b2b624a,
+     0x50a2a95d0412e289, 0x9def8df564e68581, 0xf49010a9b2e2ea5c, 0x8602ae175d9ff3f0,
+     0xbf037e245369a618, 0x8038164365f6e2b5, 0xe2e1f6163b4e8d08, 0x8df9314914f0857e},
+};
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {
+  // we aim for gaussian-like distribution of chunk sizes between min_size and max_size
+  uint64_t avg_size = (min_size + max_size) / 2;
+  // we skip calculating gearhash for the first `min_size` bytes, so we are looking for
+  // a smaller chunk as the average size
+  uint64_t target_size = avg_size - min_size;
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution
+  // `norm_factor` narrows the chunk size distribution aroun avg_size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             std::pair<uint64_t, uint64_t> size_range,
+                                             uint8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(size_range.first),
+      max_size_(size_range.second),
+      hash_mask_(GetMask(size_range.first, size_range.second, norm_factor)) {}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T value) {
+  constexpr size_t BYTE_WIDTH = sizeof(T);
+  chunk_size_ += BYTE_WIDTH;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  auto bytes = reinterpret_cast<const uint8_t*>(&value);
+  for (size_t i = 0; i < BYTE_WIDTH; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][bytes[i]];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+void ContentDefinedChunker::Roll(std::string_view value) {
+  chunk_size_ += value.size();
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (char c : value) {
+    rolling_hash_ =
+        (rolling_hash_ << 1) + GEAR_HASH_TABLE[nth_run_][static_cast<uint8_t>(c)];
+    if ((rolling_hash_ & hash_mask_) == 0) {
+      has_matched_ = true;
+    }
+  }
+}
+
+bool ContentDefinedChunker::Check() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last Check() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, not that we don't reset the rolling
+    // hash state here, so the next Check() call will continue from the current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename T>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const T& leaf_array) {
+  std::vector<Chunk> result;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int64_t offset = 0;
+    int64_t prev_offset = 0;
+    while (offset < num_levels) {
+      Roll(def_levels[offset]);
+      Roll(leaf_array.GetView(offset));
+      ++offset;
+      if (Check()) {
+        result.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    if (prev_offset < num_levels) {
+      result.emplace_back(prev_offset, prev_offset, num_levels - prev_offset);
+    }
+  } else {
+    // nested data with nulls
+    bool has_leaf_value;
+    bool is_record_boundary;
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t level_offset = 0;
+    int64_t value_offset = 0;
+    int64_t record_level_offset = 0;
+    int64_t record_value_offset = 0;
+
+    while (level_offset < num_levels) {
+      def_level = def_levels[level_offset];
+      rep_level = rep_levels[level_offset];
+
+      has_leaf_value = def_level >= level_info_.repeated_ancestor_def_level;
+      is_record_boundary = rep_level == 0;
+
+      Roll(def_level);
+      Roll(rep_level);
+      if (has_leaf_value) {
+        Roll(leaf_array.GetView(value_offset));
+      }
+
+      if (is_record_boundary && Check()) {
+        auto levels_to_write = level_offset - record_level_offset;
+        if (levels_to_write > 0) {
+          result.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+          record_level_offset = level_offset;
+          record_value_offset = value_offset;
+        }
+      }
+
+      ++level_offset;
+      if (has_leaf_value) {
+        ++value_offset;
+      }
+    }
+
+    auto levels_to_write = num_levels - record_level_offset;
+    if (levels_to_write > 0) {
+      result.emplace_back(record_level_offset, record_value_offset, levels_to_write);
+    }
+  }
+
+  return result;
+}
+
+#define PRIMITIVE_CASE(TYPE_ID, ArrowType)               \
+  case ::arrow::Type::TYPE_ID:                           \
+    return Calculate(def_levels, rep_levels, num_levels, \
+                     static_cast<const ::arrow::ArrowType##Array&>(values));
+
+const ::arrow::Result<std::vector<Chunk>> ContentDefinedChunker::GetBoundaries(
+    const int16_t* def_levels, const int16_t* rep_levels, int64_t num_levels,
+    const ::arrow::Array& values) {
+  auto type_id = values.type()->id();
+  switch (type_id) {

Review Comment:
   I tried, but it makes the implementation way more complicated. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1978941503


##########
cpp/src/parquet/column_chunker_hashtable.py:
##########
@@ -0,0 +1,90 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import hashlib
+import pathlib
+import sys
+from io import StringIO
+
+
+template = """\
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+#include <cstdint>
+
+namespace parquet {{
+namespace internal {{

Review Comment:
   Updating.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2761227255

   I added docstrings to both the C++ API and PyArrow API. and a section to the PyArrow Parquet prose page. 
   I also marked the feature experimental so we can more flexibly change it in future versions (if needed). 
   
   Since this is a non-breaking new feature marked as experimental, I would like to merge it on Monday if there is no objection so it can be included in the next release. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2019758835


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int8_t norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes
+  int64_t target_size = avg_chunk_size - min_chunk_size;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // 3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  if (norm_factor < -3 || norm_factor > 3) {
+    ARROW_LOG(WARNING) << "norm_factor=" << std::to_string(norm_factor)
+                       << " is outside the recommended range [-3, 3]";
+  }
+
+  int mask_adjustement = 3 + norm_factor;
+  int effective_bits = mask_bits - mask_adjustement;
+  if (effective_bits == 0) {
+    return 0;
+  } else if (effective_bits > 64) {
+    throw ParquetException("The number of bits in the mask cannot exceed 64, got " +
+                           std::to_string(effective_bits));
+  } else if (effective_bits < 0) {
+    throw ParquetException(
+        "The difference between min_chunk_size=" + std::to_string(min_chunk_size) +
+        " and max_chunk_size=" + std::to_string(max_chunk_size) +
+        " is too small for the given norm_factor=" + std::to_string(norm_factor) +
+        ", either increase the size range or decrease the norm_factor.");
+  } else {
+    // create the mask by setting the top mask_bits bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int8_t norm_factor)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_factor)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(const bool value) {
+    if (chunk_size_++ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+    has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+  }
+
+  template <int ByteWidth>
+  void Roll(const uint8_t* value) {
+    // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+    // true if the hash matches the mask.
+
+    chunk_size_ += ByteWidth;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (size_t i = 0; i < ByteWidth; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  template <typename T>
+  void Roll(const T* value) {
+    return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+  }
+
+  void Roll(const uint8_t* value, int64_t length) {
+    // Update the rolling hash with a binary-like value, set has_matched_ to true if the
+    // hash matches the mask.
+
+    chunk_size_ += length;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (auto i = 0; i < length; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  bool NeedNewChunk() {
+    // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+    // set to true if we encountered a match since the last NeedNewChunk() call
+    if (ARROW_PREDICT_FALSE(has_matched_)) {
+      has_matched_ = false;
+      // in order to have a normal distribution of chunk sizes, we only create a new chunk
+      // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+      // different gearhash table (gearhash's chunk size has geometric distribution, and
+      // we use central limit theorem to approximate normal distribution, see
+      // section 6.2.1 in paper https://www.cidrdb.org/cidr2023/papers/p43-low.pdf)
+      if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+        nth_run_ = 0;
+        chunk_size_ = 0;
+        return true;
+      }
+    }
+    if (ARROW_PREDICT_FALSE(chunk_size_ >= max_chunk_size_)) {
+      // we have a hard limit on the maximum chunk size, note that we don't reset the
+      // rolling hash state here, so the next NeedNewChunk() call will continue from the
+      // current state
+      chunk_size_ = 0;
+      return true;
+    }
+    return false;
+  }
+
+  void ValidateChunks(const std::vector<Chunk>& chunks, int64_t num_levels) {
+    // chunks must be non-empty and monotonic increasing
+    DCHECK(!chunks.empty());
+
+    // the first chunk must start at the first level
+    auto first_chunk = chunks.front();
+    DCHECK_EQ(first_chunk.level_offset, 0);
+    DCHECK_EQ(first_chunk.value_offset, 0);
+
+    // the following chunks must be contiguous, non-overlapping and monotonically
+    // increasing
+    auto sum_levels = first_chunk.levels_to_write;
+    for (size_t i = 1; i < chunks.size(); ++i) {
+      auto chunk = chunks[i];
+      auto prev_chunk = chunks[i - 1];
+      DCHECK_GT(chunk.levels_to_write, 0);
+      DCHECK_GE(chunk.value_offset, prev_chunk.value_offset);
+      DCHECK_EQ(chunk.level_offset, prev_chunk.level_offset + prev_chunk.levels_to_write);
+      sum_levels += chunk.levels_to_write;
+    }
+    DCHECK_EQ(sum_levels, num_levels);
+
+    // the last chunk must end at the last level
+    auto last_chunk = chunks.back();
+    DCHECK_EQ(last_chunk.level_offset + last_chunk.levels_to_write, num_levels);
+  }
+
+  template <typename RollFunc>
+  std::vector<Chunk> Calculate(const int16_t* def_levels, const int16_t* rep_levels,
+                               int64_t num_levels, const RollFunc& RollValue) {
+    // Calculate the chunk boundaries for typed Arrow arrays.
+    std::vector<Chunk> chunks;
+    int64_t offset;
+    int64_t prev_offset = 0;
+    int64_t prev_value_offset = 0;
+    bool has_def_levels = level_info_.def_level > 0;
+    bool has_rep_levels = level_info_.rep_level > 0;
+
+    if (!has_rep_levels && !has_def_levels) {
+      // fastest path for non-nested non-null data
+      for (offset = 0; offset < num_levels; ++offset) {
+        RollValue(offset);
+        if (NeedNewChunk()) {
+          chunks.push_back({prev_offset, prev_offset, offset - prev_offset});
+          prev_offset = offset;
+        }
+      }
+      // set the previous value offset to add the last chunk
+      prev_value_offset = prev_offset;
+    } else if (!has_rep_levels) {
+      // non-nested data with nulls
+      int16_t def_level;
+      for (int64_t offset = 0; offset < num_levels; ++offset) {
+        def_level = def_levels[offset];
+
+        Roll(&def_level);
+        if (def_level == level_info_.def_level) {
+          RollValue(offset);
+        }
+        if (NeedNewChunk()) {
+          chunks.push_back({prev_offset, prev_offset, offset - prev_offset});
+          prev_offset = offset;
+        }
+      }
+      // set the previous value offset to add the last chunk
+      prev_value_offset = prev_offset;
+    } else {
+      // nested data with nulls
+      int16_t def_level;
+      int16_t rep_level;
+      int64_t value_offset = 0;
+
+      for (offset = 0; offset < num_levels; ++offset) {
+        def_level = def_levels[offset];
+        rep_level = rep_levels[offset];
+
+        Roll(&def_level);
+        Roll(&rep_level);
+        if (def_level == level_info_.def_level) {
+          RollValue(value_offset);
+        }
+
+        if ((rep_level == 0) && NeedNewChunk()) {
+          // if we are at a record boundary and need a new chunk, we create a new chunk
+          auto levels_to_write = offset - prev_offset;
+          if (levels_to_write > 0) {
+            chunks.push_back({prev_offset, prev_value_offset, levels_to_write});
+            prev_offset = offset;
+            prev_value_offset = value_offset;
+          }
+        }
+        if (def_level >= level_info_.repeated_ancestor_def_level) {
+          // we only increment the value offset if we have a leaf value
+          ++value_offset;
+        }
+      }

Review Comment:
   This way I was able to have a single 
   
   ```
       // add the last chunk if we have any levels left
       if (prev_offset < num_levels) {
         chunks.push_back({prev_offset, prev_value_offset, num_levels - prev_offset});
       }
   ```
   
   at the end of the function rather than at the and of all three branches.



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int8_t norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes
+  int64_t target_size = avg_chunk_size - min_chunk_size;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // 3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  if (norm_factor < -3 || norm_factor > 3) {
+    ARROW_LOG(WARNING) << "norm_factor=" << std::to_string(norm_factor)
+                       << " is outside the recommended range [-3, 3]";
+  }
+
+  int mask_adjustement = 3 + norm_factor;
+  int effective_bits = mask_bits - mask_adjustement;
+  if (effective_bits == 0) {
+    return 0;
+  } else if (effective_bits > 64) {
+    throw ParquetException("The number of bits in the mask cannot exceed 64, got " +
+                           std::to_string(effective_bits));
+  } else if (effective_bits < 0) {
+    throw ParquetException(
+        "The difference between min_chunk_size=" + std::to_string(min_chunk_size) +
+        " and max_chunk_size=" + std::to_string(max_chunk_size) +
+        " is too small for the given norm_factor=" + std::to_string(norm_factor) +
+        ", either increase the size range or decrease the norm_factor.");
+  } else {
+    // create the mask by setting the top mask_bits bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int8_t norm_factor)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_factor)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(const bool value) {
+    if (chunk_size_++ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+    has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+  }
+
+  template <int ByteWidth>
+  void Roll(const uint8_t* value) {
+    // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+    // true if the hash matches the mask.
+
+    chunk_size_ += ByteWidth;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (size_t i = 0; i < ByteWidth; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  template <typename T>
+  void Roll(const T* value) {
+    return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+  }
+
+  void Roll(const uint8_t* value, int64_t length) {
+    // Update the rolling hash with a binary-like value, set has_matched_ to true if the
+    // hash matches the mask.
+
+    chunk_size_ += length;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (auto i = 0; i < length; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  bool NeedNewChunk() {
+    // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+    // set to true if we encountered a match since the last NeedNewChunk() call
+    if (ARROW_PREDICT_FALSE(has_matched_)) {
+      has_matched_ = false;
+      // in order to have a normal distribution of chunk sizes, we only create a new chunk
+      // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+      // different gearhash table (gearhash's chunk size has geometric distribution, and
+      // we use central limit theorem to approximate normal distribution, see
+      // section 6.2.1 in paper https://www.cidrdb.org/cidr2023/papers/p43-low.pdf)
+      if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+        nth_run_ = 0;
+        chunk_size_ = 0;
+        return true;
+      }
+    }
+    if (ARROW_PREDICT_FALSE(chunk_size_ >= max_chunk_size_)) {
+      // we have a hard limit on the maximum chunk size, note that we don't reset the
+      // rolling hash state here, so the next NeedNewChunk() call will continue from the
+      // current state
+      chunk_size_ = 0;
+      return true;
+    }
+    return false;
+  }
+
+  void ValidateChunks(const std::vector<Chunk>& chunks, int64_t num_levels) {
+    // chunks must be non-empty and monotonic increasing
+    DCHECK(!chunks.empty());
+
+    // the first chunk must start at the first level
+    auto first_chunk = chunks.front();
+    DCHECK_EQ(first_chunk.level_offset, 0);
+    DCHECK_EQ(first_chunk.value_offset, 0);
+
+    // the following chunks must be contiguous, non-overlapping and monotonically
+    // increasing
+    auto sum_levels = first_chunk.levels_to_write;
+    for (size_t i = 1; i < chunks.size(); ++i) {
+      auto chunk = chunks[i];
+      auto prev_chunk = chunks[i - 1];
+      DCHECK_GT(chunk.levels_to_write, 0);
+      DCHECK_GE(chunk.value_offset, prev_chunk.value_offset);
+      DCHECK_EQ(chunk.level_offset, prev_chunk.level_offset + prev_chunk.levels_to_write);
+      sum_levels += chunk.levels_to_write;
+    }
+    DCHECK_EQ(sum_levels, num_levels);
+
+    // the last chunk must end at the last level
+    auto last_chunk = chunks.back();
+    DCHECK_EQ(last_chunk.level_offset + last_chunk.levels_to_write, num_levels);
+  }
+
+  template <typename RollFunc>
+  std::vector<Chunk> Calculate(const int16_t* def_levels, const int16_t* rep_levels,
+                               int64_t num_levels, const RollFunc& RollValue) {
+    // Calculate the chunk boundaries for typed Arrow arrays.
+    std::vector<Chunk> chunks;
+    int64_t offset;
+    int64_t prev_offset = 0;
+    int64_t prev_value_offset = 0;
+    bool has_def_levels = level_info_.def_level > 0;
+    bool has_rep_levels = level_info_.rep_level > 0;
+
+    if (!has_rep_levels && !has_def_levels) {
+      // fastest path for non-nested non-null data
+      for (offset = 0; offset < num_levels; ++offset) {
+        RollValue(offset);
+        if (NeedNewChunk()) {
+          chunks.push_back({prev_offset, prev_offset, offset - prev_offset});
+          prev_offset = offset;
+        }
+      }
+      // set the previous value offset to add the last chunk
+      prev_value_offset = prev_offset;
+    } else if (!has_rep_levels) {
+      // non-nested data with nulls
+      int16_t def_level;
+      for (int64_t offset = 0; offset < num_levels; ++offset) {
+        def_level = def_levels[offset];
+
+        Roll(&def_level);
+        if (def_level == level_info_.def_level) {
+          RollValue(offset);
+        }
+        if (NeedNewChunk()) {
+          chunks.push_back({prev_offset, prev_offset, offset - prev_offset});
+          prev_offset = offset;
+        }
+      }
+      // set the previous value offset to add the last chunk
+      prev_value_offset = prev_offset;
+    } else {
+      // nested data with nulls
+      int16_t def_level;
+      int16_t rep_level;
+      int64_t value_offset = 0;
+
+      for (offset = 0; offset < num_levels; ++offset) {
+        def_level = def_levels[offset];
+        rep_level = rep_levels[offset];
+
+        Roll(&def_level);
+        Roll(&rep_level);
+        if (def_level == level_info_.def_level) {
+          RollValue(value_offset);
+        }
+
+        if ((rep_level == 0) && NeedNewChunk()) {
+          // if we are at a record boundary and need a new chunk, we create a new chunk
+          auto levels_to_write = offset - prev_offset;
+          if (levels_to_write > 0) {
+            chunks.push_back({prev_offset, prev_value_offset, levels_to_write});
+            prev_offset = offset;
+            prev_value_offset = value_offset;
+          }
+        }
+        if (def_level >= level_info_.repeated_ancestor_def_level) {
+          // we only increment the value offset if we have a leaf value
+          ++value_offset;
+        }
+      }

Review Comment:
   No, the other two branches are not incrementing `prev_value_offset` separately to the level offsets se we set `prev_value_offset = prev_[level]_offset` at those branches. The last branch handles the nested case and it maintains `prev_value_offset` already. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["mapleFU (via GitHub)" <gi...@apache.org>]

mapleFU commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2016980045


##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1382,11 +1383,15 @@ class TypedColumnWriterImpl : public ColumnWriterImpl,
                                          chunk.levels_to_write, *chunk_array, ctx,
                                          maybe_parent_nulls));
         }
-        if (num_buffered_values_ > 0) {
+        bool is_last_chunk = i == chunks.size() - 1;
+        if (num_buffered_values_ > 0 && !is_last_chunk) {
           // Explicitly add a new data page according to the content-defined chunk
           // boundaries. This way the same chunks will have the same byte-sequence
           // in the resulting file, which can be identified by content addressible
           // storage.
+          // Note that the last chunk doesn't trigger a new data page in order to
+          // allow subsequent WriteArrow() calls to continue writing to the same
+          // data page, the chunker's state is not being reset after the last chunk.

Review Comment:
   Assume a scenerio, when feed last chunk, what if `["a", ..., long-final string]`, long-final string's size exceeds maximum chunk size. What would happen when long-final string is solely in final chunk and not being flushed?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2017707076


##########
cpp/src/parquet/schema.cc:
##########
@@ -28,6 +28,7 @@
 #include "parquet/exception.h"
 #include "parquet/schema_internal.h"
 #include "parquet/thrift_internal.h"
+#include "parquet/windows_fixup.h"  // for OPTIONAL

Review Comment:
   I had to add the windows fixup here to overcome 
   
   ```
   In file included from D:/a/arrow/arrow/build/cpp/src/parquet/CMakeFiles/parquet_shared.dir/Unity/unity_3_cxx.cxx:16:
   D:/a/arrow/arrow/cpp/src/parquet/schema.cc: In function 'void parquet::schema::PrintRepLevel(parquet::Repetition::type, std::ostream&)':
   D:/a/arrow/arrow/cpp/src/parquet/schema.cc:630:30: error: expected unqualified-id before ':' token
     630 |     case Repetition::OPTIONAL:
   ```
   
   https://github.com/apache/arrow/actions/runs/14115826826/job/39545693922?pr=45360#step:7:838
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2017639249


##########
cpp/src/parquet/schema.cc:
##########
@@ -25,6 +25,7 @@
 #include <utility>
 
 #include "arrow/util/logging.h"
+#include "arrow/util/windows_fixup.h"  // for OPTIONAL

Review Comment:
   I had to add the windows fixup here to overcome 
   
   ```
   In file included from D:/a/arrow/arrow/build/cpp/src/parquet/CMakeFiles/parquet_shared.dir/Unity/unity_3_cxx.cxx:16:
   D:/a/arrow/arrow/cpp/src/parquet/schema.cc: In function 'void parquet::schema::PrintRepLevel(parquet::Repetition::type, std::ostream&)':
   D:/a/arrow/arrow/cpp/src/parquet/schema.cc:630:30: error: expected unqualified-id before ':' token
     630 |     case Repetition::OPTIONAL:
   ```
   
   https://github.com/apache/arrow/actions/runs/14115826826/job/39545693922?pr=45360#step:7:838
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025068728


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int8_t norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes
+  int64_t target_size = avg_chunk_size - min_chunk_size;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // 3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  if (norm_factor < -3 || norm_factor > 3) {
+    ARROW_LOG(WARNING) << "norm_factor=" << norm_factor
+                       << " is outside the recommended range [-3, 3]";
+  }
+
+  int mask_adjustement = 3 + norm_factor;
+  int effective_bits = mask_bits - mask_adjustement;
+  if (effective_bits == 0) {
+    return 0;
+  } else if (effective_bits > 64) {
+    throw ParquetException("The number of bits in the mask cannot exceed 64, got " +
+                           std::to_string(effective_bits));
+  } else if (effective_bits < 0) {
+    throw ParquetException(
+        "The difference between min_chunk_size=" + std::to_string(min_chunk_size) +
+        " and max_chunk_size=" + std::to_string(max_chunk_size) +
+        " is too small for the given norm_factor=" + std::to_string(norm_factor) +
+        ", either increase the size range or decrease the norm_factor.");
+  } else {
+    // create the mask by setting the top mask_bits bits

Review Comment:
   Removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025097329


##########
cpp/src/parquet/properties.h:
##########
@@ -245,6 +245,14 @@ class PARQUET_EXPORT ColumnProperties {
   bool page_index_enabled_;
 };
 
+struct CDCOptions {
+  int64_t min_chunk_size;
+  int64_t max_chunk_size;
+  int8_t norm_factor;
+};
+
+static constexpr CDCOptions kDefaultCdcOptions = CDCOptions{256 * 1024, 1024 * 1024, 0};

Review Comment:
   Using `CDCOptions`, let me know if I should prefer the other.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025150264


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>

Review Comment:
   Done. There are other tests files with this order as well. Would be nice to have something like `isort`, it is pretty easy to miss.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025175912


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1992134292


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,319 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_size, max_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// probability of matching and switching them based on the actual chunk size), we rather
+/// use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_size The minimum chunk size (default 256KiB)
+// @param max_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t GetMask(int64_t min_size, int64_t max_size, uint8_t norm_factor) {
+  // calculate the average size of the chunks
+  int64_t avg_size = (min_size + max_size) / 2;
+  // since we are skipping the first `min_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_size` bytes
+  int64_t target_size = avg_size - min_size;
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the log2 of the target size
+  size_t mask_bits = static_cast<size_t>(std::floor(std::log2(target_size)));
+  // -3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size
+  size_t effective_bits = mask_bits - 3 - norm_factor;
+  return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+}
+
+ContentDefinedChunker::ContentDefinedChunker(const LevelInfo& level_info,
+                                             int64_t min_size, int64_t max_size,
+                                             int8_t norm_factor)
+    : level_info_(level_info),
+      min_size_(min_size),
+      max_size_(max_size),
+      hash_mask_(GetMask(min_size, max_size, norm_factor)) {
+  if (min_size_ < 0) {
+    throw ParquetException("min_size must be non-negative");
+  }
+  if (max_size_ < 0) {
+    throw ParquetException("max_size must be non-negative");
+  }
+  if (min_size_ > max_size_) {
+    throw ParquetException("min_size must be less than or equal to max_size");
+  }
+}
+
+void ContentDefinedChunker::Roll(const bool value) {
+  if (chunk_size_++ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+  has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+}
+
+template <int ByteWidth>
+void ContentDefinedChunker::Roll(const uint8_t* value) {
+  chunk_size_ += ByteWidth;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (size_t i = 0; i < ByteWidth; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+template <typename T>
+void ContentDefinedChunker::Roll(const T* value) {
+  return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+}
+
+void ContentDefinedChunker::Roll(const uint8_t* value, int64_t length) {
+  chunk_size_ += length;
+  if (chunk_size_ < min_size_) {
+    // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+    // chunking process since the gearhash doesn't need to be updated
+    return;
+  }
+  for (auto i = 0; i < length; ++i) {
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+    has_matched_ = has_matched_ || ((rolling_hash_ & hash_mask_) == 0);
+  }
+}
+
+bool ContentDefinedChunker::NeedNewChunk() {
+  // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+  // set to true if we encountered a match since the last NeedNewChunk() call
+  if (ARROW_PREDICT_FALSE(has_matched_)) {
+    has_matched_ = false;
+    // in order to have a normal distribution of chunk sizes, we only create a new chunk
+    // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+    // different gearhash table (gearhash's chunk size has exponential distribution, and
+    // we use central limit theorem to approximate normal distribution)
+    if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+      nth_run_ = 0;
+      chunk_size_ = 0;
+      return true;
+    }
+  }
+  if (ARROW_PREDICT_FALSE(chunk_size_ >= max_size_)) {
+    // we have a hard limit on the maximum chunk size, note that we don't reset the
+    // rolling hash state here, so the next NeedNewChunk() call will continue from the
+    // current state
+    chunk_size_ = 0;
+    return true;
+  }
+  return false;
+}
+
+template <typename RollFunc>
+const std::vector<Chunk> ContentDefinedChunker::Calculate(const int16_t* def_levels,
+                                                          const int16_t* rep_levels,
+                                                          int64_t num_levels,
+                                                          const RollFunc& RollValue) {
+  std::vector<Chunk> chunks;
+  int64_t offset;
+  int64_t prev_offset = 0;
+  int64_t prev_value_offset = 0;
+  bool has_def_levels = level_info_.def_level > 0;
+  bool has_rep_levels = level_info_.rep_level > 0;
+
+  if (!has_rep_levels && !has_def_levels) {
+    // fastest path for non-nested non-null data
+    for (offset = 0; offset < num_levels; ++offset) {
+      RollValue(offset);
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else if (!has_rep_levels) {
+    // non-nested data with nulls
+    int16_t def_level;
+    for (int64_t offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+
+      Roll(&def_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(offset);
+      }
+      if (NeedNewChunk()) {
+        chunks.emplace_back(prev_offset, prev_offset, offset - prev_offset);
+        prev_offset = offset;
+      }
+    }
+    // set the previous value offset to add the last chunk
+    prev_value_offset = prev_offset;
+  } else {
+    // nested data with nulls
+    int16_t def_level;
+    int16_t rep_level;
+    int64_t value_offset = 0;
+
+    for (offset = 0; offset < num_levels; ++offset) {
+      def_level = def_levels[offset];
+      rep_level = rep_levels[offset];
+
+      Roll(&def_level);
+      Roll(&rep_level);
+      if (def_level == level_info_.def_level) {
+        RollValue(value_offset);
+      }
+
+      if ((rep_level == 0) && NeedNewChunk()) {
+        // if we are at a record boundary and need a new chunk, we create a new chunk
+        auto levels_to_write = offset - prev_offset;
+        if (levels_to_write > 0) {
+          chunks.emplace_back(prev_offset, prev_value_offset, levels_to_write);
+          prev_offset = offset;
+          prev_value_offset = value_offset;
+        }
+      }
+      if (def_level >= level_info_.repeated_ancestor_def_level) {
+        // we only increment the value offset if we have a leaf value
+        ++value_offset;
+      }
+    }
+  }
+
+  // add the last chunk if we have any levels left
+  if (prev_offset < num_levels) {
+    chunks.emplace_back(prev_offset, prev_value_offset, num_levels - prev_offset);
+  }
+  return chunks;
+}
+
+#define FIXED_WIDTH_CASE(ByteWidth)                                       \
+  {                                                                       \
+    const auto raw_values = values.data()->GetValues<uint8_t>(1);         \

Review Comment:
   Isn't `GetValues()` with a single argument supposed to incorporate the offset?
   
   ```cpp
    // Access a buffer's data as a typed C pointer
     template <typename T>
     inline const T* GetValues(int i, int64_t absolute_offset) const {
       if (buffers[i]) {
         return reinterpret_cast<const T*>(buffers[i]->data()) + absolute_offset;
       } else {
         return NULLPTR;
       }
     }
   
     template <typename T>
     inline const T* GetValues(int i) const {
       return GetValues<T>(i, offset);
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025648985


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();
+    auto min_length = ElementCount(min_chunk_size, byte_width, nullable);
+    auto max_length = ElementCount(max_chunk_size, byte_width, nullable);
+    AssertAllBetween(base_info.page_lengths, min_length, max_length);
+    AssertAllBetween(modified_info.page_lengths, min_length, max_length);
+  } else if (::arrow::is_base_binary_like(dtype->id()) && !enable_dictionary) {
+    AssertAllBetween(base_info.page_sizes, min_chunk_size, max_chunk_size);
+    AssertAllBetween(modified_info.page_sizes, min_chunk_size, max_chunk_size);
+  }
+}
+
+constexpr int64_t kMinChunkSize = 8 * 1024;
+constexpr int64_t kMaxChunkSize = 32 * 1024;
+constexpr int64_t kPartSize = 128 * 1024;
+constexpr int64_t kEditSize = 128;
+
+struct CaseConfig {
+  // Arrow data type to generate the testing data for
+  std::shared_ptr<::arrow::DataType> dtype;
+  // Whether the data type is nullable
+  bool is_nullable;
+  // Approximate number of bytes per record to calculate the number of elements to
+  // generate
+  size_t bytes_per_record;
+  // Data page version to use
+  ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1;
+};
+
+// Define PrintTo for MyStruct
+void PrintTo(const CaseConfig& param, std::ostream* os) {
+  *os << "{ " << param.dtype->ToString();
+  if (param.is_nullable) {
+    *os << " nullable";
+  }
+  *os << " }";
+}
+
+class TestCDC : public ::testing::Test {
+ public:
+  uint64_t GetMask(const ContentDefinedChunker& cdc) const { return cdc.GetMask(); }
+};
+
+TEST_F(TestCDC, RollingHashMaskCalculation) {
+  auto le = LevelInfo();
+  auto min_size = 256 * 1024;
+  auto max_size = 1024 * 1024;
+
+  auto cdc0 = ContentDefinedChunker(le, min_size, max_size, 0);
+  ASSERT_EQ(GetMask(cdc0), 0xFFFE000000000000);
+
+  auto cdc1 = ContentDefinedChunker(le, min_size, max_size, 1);
+  ASSERT_EQ(GetMask(cdc1), 0xFFFC000000000000);
+
+  auto cdc2 = ContentDefinedChunker(le, min_size, max_size, 2);
+  ASSERT_EQ(GetMask(cdc2), 0xFFF8000000000000);
+
+  auto cdc3 = ContentDefinedChunker(le, min_size, max_size, 3);
+  ASSERT_EQ(GetMask(cdc3), 0xFFF0000000000000);
+
+  auto cdc4 = ContentDefinedChunker(le, min_size, max_size, -1);
+  ASSERT_EQ(GetMask(cdc4), 0xFFFF000000000000);
+
+  // this is the smallest possible mask always matching, by using 8 hashtables
+  // we are going to have a match every 8 bytes; this is an unrealistic case
+  // but checking for the correctness of the mask calculation
+  auto cdc5 = ContentDefinedChunker(le, 0, 16, 0);
+  ASSERT_EQ(GetMask(cdc5), 0x0000000000000000);
+
+  auto cdc6 = ContentDefinedChunker(le, 0, 32, 1);
+  ASSERT_EQ(GetMask(cdc6), 0x0000000000000000);
+
+  auto cdc7 = ContentDefinedChunker(le, 0, 16, -1);
+  ASSERT_EQ(GetMask(cdc7), 0x8000000000000000);
+
+  // another unrealistic case, checking for the validation
+  auto cdc8 = ContentDefinedChunker(le, 128, 384, -60);
+  ASSERT_EQ(GetMask(cdc8), 0xFFFFFFFFFFFFFFFF);
+}
+
+TEST_F(TestCDC, WriteSingleColumnParquetFile) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  std::vector<int32_t> numbers = {1, 2, 3, 4, 5};
+  std::vector<uint8_t> valid_bits = {1, 0, 1, 0, 1};
+  EXPECT_THROW(
+      int_column_writer.WriteBatch(numbers.size(), nullptr, nullptr, numbers.data()),
+      ParquetException);
+  EXPECT_THROW(int_column_writer.WriteBatchSpaced(numbers.size(), nullptr, nullptr,
+                                                  valid_bits.data(), 0, numbers.data()),
+               ParquetException);
+}
+
+TEST_F(TestCDC, LastChunkDoesntTriggerAddDataPage) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()
+                   ->content_defined_chunking_options(kMinChunkSize, kMaxChunkSize, 0)
+                   ->disable_dictionary()
+                   ->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  ASSERT_OK_AND_ASSIGN(auto array, ::arrow::gen::Step()->Generate(8000));
+  auto arrow_props = default_arrow_writer_properties();
+  auto arrow_ctx = ArrowWriteContext(default_memory_pool(), arrow_props.get());
+
+  // Calling WriteArrow twice, we expect that the first call doesn't add a new data page
+  // at the end allowing subsequent calls to append to the same page
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+
+  int_column_writer.Close();
+  writer->Close();
+  ASSERT_OK_AND_ASSIGN(auto buffer, sink->Finish());
+
+  auto info = GetColumnParquetInfo(buffer);
+  ASSERT_EQ(info.size(), 1);
+
+  // AssertAllBetween allow the last chunk size to be smaller than the min_chunk_size
+  AssertAllBetween(info[0].page_sizes, kMinChunkSize, kMaxChunkSize);
+  AssertAllBetween(info[0].page_lengths, 3000, 5000);
+}
+
+TEST_F(TestCDC, ChunkSizeParameterValidation) {
+  // Test that constructor validates min/max chunk size parameters
+  auto li = LevelInfo();
+
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 256 * 1024, 1024 * 1024));
+
+  // with norm_factor=0 the difference between min and max chunk size must be
+  // at least 16
+  ASSERT_THROW(ContentDefinedChunker(li, 0, -1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1024, 512), ParquetException);
+
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16));
+  ASSERT_THROW(ContentDefinedChunker(li, -16, -16), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 16, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 32, 32), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 32, 48));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1024 * 1024, 2 * 1024 * 1024));
+  ASSERT_NO_THROW(
+      ContentDefinedChunker(li, 1024 * 1024 * 1024L, 2LL * 1024 * 1024 * 1024L));
+
+  // with norm_factor=1 the difference between min and max chunk size must be
+  // at least 64
+  ASSERT_THROW(ContentDefinedChunker(li, 1, -1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 32, 1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1, 33, 1));
+
+  // with norm_factor=2 the difference between min and max chunk size must be
+  // at least 128
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 63, 2), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 64, 2));
+
+  // with norm_factor=-1 the difference between min and max chunk size must be
+  // at least 8
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 7, -1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 8, -1));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16, -1));
+
+  // test the norm_factor extremes
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0, -68), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 1, -67));
+  ASSERT_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 59),
+               ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 58));
+}
+
+class TestCDCSingleRowGroup : public ::testing::TestWithParam<CaseConfig> {
+ protected:
+  // Column random table parts for testing
+  std::shared_ptr<Field> field_;
+  std::shared_ptr<Table> part1_, part2_, part3_, part4_, part5_, part6_, part7_;
+
+  void SetUp() override {
+    const auto& param = GetParam();
+    auto field_ = ::arrow::field("f0", param.dtype, param.is_nullable);
+    auto schema = ::arrow::schema({field_});
+
+    auto part_length = kPartSize / param.bytes_per_record;
+    auto edit_length = kEditSize / param.bytes_per_record;
+    ASSERT_OK_AND_ASSIGN(part1_, GenerateTable(schema, part_length, 0));
+    ASSERT_OK_AND_ASSIGN(part2_, GenerateTable(schema, edit_length, 1));
+    ASSERT_OK_AND_ASSIGN(part3_, GenerateTable(schema, part_length, part_length));
+    ASSERT_OK_AND_ASSIGN(part4_, GenerateTable(schema, edit_length, 2));
+    ASSERT_OK_AND_ASSIGN(part5_, GenerateTable(schema, part_length, 2 * part_length));
+    ASSERT_OK_AND_ASSIGN(part6_, GenerateTable(schema, edit_length, 3));
+    ASSERT_OK_AND_ASSIGN(part7_, GenerateTable(schema, edit_length, 4));
+  }
+};
+
+TEST_P(TestCDCSingleRowGroup, DeleteOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/0,
+                                /*exact_number_of_smaller_diffs=*/1, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, DeleteTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part3_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/0,
+                                /*exact_number_of_smaller_diffs=*/2, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, UpdateOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part4_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*max_number_of_equal_diffs=*/1);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, UpdateTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified,
+                       ConcatAndCombine({part1_, part6_, part3_, part7_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*max_number_of_equal_diffs=*/2);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, InsertOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/1,
+                                /*exact_number_of_smaller_diffs=*/0, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, InsertTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part3_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/2,
+                                /*exact_number_of_smaller_diffs=*/0, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, Append) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part2_, part3_, part4_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto original_page_lengths = base_info.front().page_lengths;
+    auto modified_page_lengths = modified_info.front().page_lengths;
+    ASSERT_GE(original_page_lengths.size(), modified_page_lengths.size());

Review Comment:
   Nice catch. If I change the part/append size it actually breaks, so the assertion was wrong. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2026611843


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();

Review Comment:
   Well, this is trying to estimate the number of boolean values from the data page size, right? I don't call it an implementation detail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1983551683


##########
cpp/src/parquet/chunker_internal_hashtable.h:
##########
@@ -0,0 +1,545 @@
+// Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   Sure, updating.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1983591472


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+
+constexpr uint64_t GEAR_HASH_TABLE[8][256] = {

Review Comment:
   `With norm_factor=2`
   
   ![newplot-12](https://github.com/user-attachments/assets/3ba62312-fb85-4a2b-b704-dea0ff75d1fc)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1983892222


##########
cpp/src/parquet/chunker_internal_hashtable.py:
##########
@@ -0,0 +1,90 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import hashlib
+import pathlib
+import sys
+from io import StringIO
+
+
+template = """\
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+#include <cstdint>
+
+namespace parquet {{
+namespace internal {{
+
+constexpr uint64_t GEARHASH_TABLE[8][256] = {{
+{content}}};
+
+}}  // namespace internal
+}}  // namespace parquet
+"""
+
+
+def generate_hash(n: int, seed: int):
+    value = bytes([seed] * 64 + [n] * 64)
+    hasher = hashlib.md5(value)
+    return hasher.hexdigest()[:16]
+
+
+def generate_hashtable(seed: int, length=256, comma=True):

Review Comment:
   Remnant, removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1983981980


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,908 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+#define GENERATE_CASE_BODY(BUILDER_TYPE, VALUE_EXPR)   \

Review Comment:
   Replaced with a templated `GenerateArray()` function.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1981315325


##########
cpp/src/parquet/column_chunker.cc:
##########
@@ -0,0 +1,778 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/column_chunker.h"
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet {
+namespace internal {
+

Review Comment:
   Added references to the original `FastCDC` paper and to another describing the chunk size normalization approach.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1963656555


##########
cpp/src/parquet/column_chunker.h:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+using arrow::internal::checked_cast;
+
+namespace parquet {
+namespace internal {
+
+// Constants
+const uint64_t GEAR_HASH_TABLE[] = {
+    0xb088d3a9e840f559, 0x5652c7f739ed20d6, 0x45b28969898972ab, 0x6b0a89d5b68ec777,
+    0x368f573e8b7a31b7, 0x1dc636dce936d94b, 0x207a4c4e5554d5b6, 0xa474b34628239acb,
+    0x3b06a83e1ca3b912, 0x90e78d6c2f02baf7, 0xe1c92df7150d9a8a, 0x8e95053a1086d3ad,
+    0x5a2ef4f1b83a0722, 0xa50fac949f807fae, 0x0e7303eb80d8d681, 0x99b07edc1570ad0f,
+    0x689d2fb555fd3076, 0x00005082119ea468, 0xc4b08306a88fcc28, 0x3eb0678af6374afd,
+    0xf19f87ab86ad7436, 0xf2129fbfbe6bc736, 0x481149575c98a4ed, 0x0000010695477bc5,
+    0x1fba37801a9ceacc, 0x3bf06fd663a49b6d, 0x99687e9782e3874b, 0x79a10673aa50d8e3,
+    0xe4accf9e6211f420, 0x2520e71f87579071, 0x2bd5d3fd781a8a9b, 0x00de4dcddd11c873,
+    0xeaa9311c5a87392f, 0xdb748eb617bc40ff, 0xaf579a8df620bf6f, 0x86a6e5da1b09c2b1,
+    0xcc2fc30ac322a12e, 0x355e2afec1f74267, 0x2d99c8f4c021a47b, 0xbade4b4a9404cfc3,
+    0xf7b518721d707d69, 0x3286b6587bf32c20, 0x0000b68886af270c, 0xa115d6e4db8a9079,
+    0x484f7e9c97b2e199, 0xccca7bb75713e301, 0xbf2584a62bb0f160, 0xade7e813625dbcc8,
+    0x000070940d87955a, 0x8ae69108139e626f, 0xbd776ad72fde38a2, 0xfb6b001fc2fcc0cf,
+    0xc7a474b8e67bc427, 0xbaf6f11610eb5d58, 0x09cb1f5b6de770d1, 0xb0b219e6977d4c47,
+    0x00ccbc386ea7ad4a, 0xcc849d0adf973f01, 0x73a3ef7d016af770, 0xc807d2d386bdbdfe,
+    0x7f2ac9966c791730, 0xd037a86bc6c504da, 0xf3f17c661eaa609d, 0xaca626b04daae687,
+    0x755a99374f4a5b07, 0x90837ee65b2caede, 0x6ee8ad93fd560785, 0x0000d9e11053edd8,
+    0x9e063bb2d21cdbd7, 0x07ab77f12a01d2b2, 0xec550255e6641b44, 0x78fb94a8449c14c6,
+    0xc7510e1bc6c0f5f5, 0x0000320b36e4cae3, 0x827c33262c8b1a2d, 0x14675f0b48ea4144,
+    0x267bd3a6498deceb, 0xf1916ff982f5035e, 0x86221b7ff434fb88, 0x9dbecee7386f49d8,
+    0xea58f8cac80f8f4a, 0x008d198692fc64d8, 0x6d38704fbabf9a36, 0xe032cb07d1e7be4c,
+    0x228d21f6ad450890, 0x635cb1bfc02589a5, 0x4620a1739ca2ce71, 0xa7e7dfe3aae5fb58,
+    0x0c10ca932b3c0deb, 0x2727fee884afed7b, 0xa2df1c6df9e2ab1f, 0x4dcdd1ac0774f523,
+    0x000070ffad33e24e, 0xa2ace87bc5977816, 0x9892275ab4286049, 0xc2861181ddf18959,
+    0xbb9972a042483e19, 0xef70cd3766513078, 0x00000513abfc9864, 0xc058b61858c94083,
+    0x09e850859725e0de, 0x9197fb3bf83e7d94, 0x7e1e626d12b64bce, 0x520c54507f7b57d1,
+    0xbee1797174e22416, 0x6fd9ac3222e95587, 0x0023957c9adfbf3e, 0xa01c7d7e234bbe15,
+    0xaba2c758b8a38cbb, 0x0d1fa0ceec3e2b30, 0x0bb6a58b7e60b991, 0x4333dd5b9fa26635,
+    0xc2fd3b7d4001c1a3, 0xfb41802454731127, 0x65a56185a50d18cb, 0xf67a02bd8784b54f,
+    0x696f11dd67e65063, 0x00002022fca814ab, 0x8cd6be912db9d852, 0x695189b6e9ae8a57,
+    0xee9453b50ada0c28, 0xd8fc5ea91a78845e, 0xab86bf191a4aa767, 0x0000c6b5c86415e5,
+    0x267310178e08a22e, 0xed2d101b078bca25, 0x3b41ed84b226a8fb, 0x13e622120f28dc06,
+    0xa315f5ebfb706d26, 0x8816c34e3301bace, 0xe9395b9cbb71fdae, 0x002ce9202e721648,
+    0x4283db1d2bb3c91c, 0xd77d461ad2b1a6a5, 0xe2ec17e46eeb866b, 0xb8e0be4039fbc47c,
+    0xdea160c4d5299d04, 0x7eec86c8d28c3634, 0x2119ad129f98a399, 0xa6ccf46b61a283ef,
+    0x2c52cedef658c617, 0x2db4871169acdd83, 0x0000f0d6f39ecbe9, 0x3dd5d8c98d2f9489,
+    0x8a1872a22b01f584, 0xf282a4c40e7b3cf2, 0x8020ec2ccb1ba196, 0x6693b6e09e59e313,
+    0x0000ce19cc7c83eb, 0x20cb5735f6479c3b, 0x762ebf3759d75a5b, 0x207bfe823d693975,
+    0xd77dc112339cd9d5, 0x9ba7834284627d03, 0x217dc513e95f51e9, 0xb27b1a29fc5e7816,
+    0x00d5cd9831bb662d, 0x71e39b806d75734c, 0x7e572af006fb1a23, 0xa2734f2f6ae91f85,
+    0xbf82c6b5022cddf2, 0x5c3beac60761a0de, 0xcdc893bb47416998, 0x6d1085615c187e01,
+    0x77f8ae30ac277c5d, 0x917c6b81122a2c91, 0x5b75b699add16967, 0x0000cf6ae79a069b,
+    0xf3c40afa60de1104, 0x2063127aa59167c3, 0x621de62269d1894d, 0xd188ac1de62b4726,
+    0x107036e2154b673c, 0x0000b85f28553a1d, 0xf2ef4e4c18236f3d, 0xd9d6de6611b9f602,
+    0xa1fc7955fb47911c, 0xeb85fd032f298dbd, 0xbe27502fb3befae1, 0xe3034251c4cd661e,
+    0x441364d354071836, 0x0082b36c75f2983e, 0xb145910316fa66f0, 0x021c069c9847caf7,
+    0x2910dfc75a4b5221, 0x735b353e1c57a8b5, 0xce44312ce98ed96c, 0xbc942e4506bdfa65,
+    0xf05086a71257941b, 0xfec3b215d351cead, 0x00ae1055e0144202, 0xf54b40846f42e454,
+    0x00007fd9c8bcbcc8, 0xbfbd9ef317de9bfe, 0xa804302ff2854e12, 0x39ce4957a5e5d8d4,
+    0xffb9e2a45637ba84, 0x55b9ad1d9ea0818b, 0x00008acbf319178a, 0x48e2bfc8d0fbfb38,
+    0x8be39841e848b5e8, 0x0e2712160696a08b, 0xd51096e84b44242a, 0x1101ba176792e13a,
+    0xc22e770f4531689d, 0x1689eff272bbc56c, 0x00a92a197f5650ec, 0xbc765990bda1784e,
+    0xc61441e392fcb8ae, 0x07e13a2ced31e4a0, 0x92cbe984234e9d4d, 0x8f4ff572bb7d8ac5,
+    0x0b9670c00b963bd0, 0x62955a581a03eb01, 0x645f83e5ea000254, 0x41fce516cd88f299,
+    0xbbda9748da7a98cf, 0x0000aab2fe4845fa, 0x19761b069bf56555, 0x8b8f5e8343b6ad56,
+    0x3e5d1cfd144821d9, 0xec5c1e2ca2b0cd8f, 0xfaf7e0fea7fbb57f, 0x000000d3ba12961b,
+    0xda3f90178401b18e, 0x70ff906de33a5feb, 0x0527d5a7c06970e7, 0x22d8e773607c13e9,
+    0xc9ab70df643c3bac, 0xeda4c6dc8abe12e3, 0xecef1f410033e78a, 0x0024c2b274ac72cb,
+    0x06740d954fa900b4, 0x1d7a299b323d6304, 0xb3c37cb298cbead5, 0xc986e3c76178739b,
+    0x9fabea364b46f58a, 0x6da214c5af85cc56, 0x17a43ed8b7a38f84, 0x6eccec511d9adbeb,
+    0xf9cab30913335afb, 0x4a5e60c5f415eed2, 0x00006967503672b4, 0x9da51d121454bb87,
+    0x84321e13b9bbc816, 0xfb3d6fb6ab2fdd8d, 0x60305eed8e160a8d, 0xcbbf4b14e9946ce8,
+    0x00004f63381b10c3, 0x07d5b7816fcc4e10, 0xe5a536726a6a8155, 0x57afb23447a07fdd,
+    0x18f346f7abc9d394, 0x636dc655d61ad33d, 0xcc8bab4939f7f3f6, 0x63c7a906c1dd187b};
+
+const uint64_t MASK = 0xffff00000000000;
+// const int MIN_LEN = 65536 / 8;
+// const int MAX_LEN = 65536 * 2;
+const int64_t MIN_LEN = 256 * 1024;

Review Comment:
   Now the user provides the min and max values then the chunker calculates the mask size trying to achieve reasonably uniform distribution between the two extremes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2016989421


##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1382,11 +1383,15 @@ class TypedColumnWriterImpl : public ColumnWriterImpl,
                                          chunk.levels_to_write, *chunk_array, ctx,
                                          maybe_parent_nulls));
         }
-        if (num_buffered_values_ > 0) {
+        bool is_last_chunk = i == chunks.size() - 1;
+        if (num_buffered_values_ > 0 && !is_last_chunk) {
           // Explicitly add a new data page according to the content-defined chunk
           // boundaries. This way the same chunks will have the same byte-sequence
           // in the resulting file, which can be identified by content addressible
           // storage.
+          // Note that the last chunk doesn't trigger a new data page in order to
+          // allow subsequent WriteArrow() calls to continue writing to the same
+          // data page, the chunker's state is not being reset after the last chunk.

Review Comment:
   The chunker post-checks max size after adding the value to the chunk, if it exceeds the max chunk size then we don't add more values to that chunk but rather split. This is similar to how `max_data_page_size` is being checked.
   
   We are just not calling `AddDataPage()` after the last chunk, but that "ongoing" data page will be flushed nonetheless when the writer gets closed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r1988115020


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,250 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cmath>
+#include <string>
+#include <vector>
+#include "arrow/array.h"
+#include "arrow/util/logging.h"
+#include "parquet/chunker_internal_hashtable.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// create a fake null array class with a GetView method returning 0 always
+class FakeNullArray {
+ public:
+  uint8_t GetView(int64_t i) const { return 0; }
+
+  std::shared_ptr<::arrow::DataType> type() const { return ::arrow::null(); }
+
+  int64_t null_count() const { return 0; }
+};
+
+static uint64_t GetMask(uint64_t min_size, uint64_t max_size, uint8_t norm_factor) {

Review Comment:
   Yes, added a more detailed description.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086155388


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,412 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+namespace {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_level parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_level Normalization level (default 0)
+// @return The mask used to compare against the rolling hash
+uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size, int norm_level) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be non-negative");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes; also divide by the number of gearhash tables to have a
+  // a more gaussian-like distribution
+  int64_t target_size = (avg_chunk_size - min_chunk_size) / kNumGearhashTables;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // a user defined `norm_level` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_level we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_level is 0 by
+  // default
+  int effective_bits = mask_bits - norm_level;
+
+  if (effective_bits < 1 || effective_bits > 63) {
+    throw ParquetException(
+        "The number of bits in the CDC mask must be between 1 and 63, got " +
+        std::to_string(effective_bits));
+  } else {
+    // create the mask by setting the top bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+}  // namespace
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int norm_level)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_level)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the

Review Comment:
   It depends what the goal is. If only informative, then we don't need to serialize everything (there are so many parameters to play with). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086156941


##########
cpp/src/parquet/properties.h:
##########
@@ -245,6 +245,34 @@ class PARQUET_EXPORT ColumnProperties {
   bool page_index_enabled_;
 };
 
+// EXPERIMENTAL: Options for content-defined chunking.
+struct PARQUET_EXPORT CdcOptions {
+  /// Minimum chunk size in bytes, default is 256 KiB

Review Comment:
   Yes, adding it to the comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2875501508

   Revision: 9275cadd5d730f09f32fd58592d62310c63c3831
   
   Submitted crossbow builds: [ursacomputing/crossbow @ actions-b8b690defb](https://github.com/ursacomputing/crossbow/branches/all?query=actions-b8b690defb)
   
   |Task|Status|
   |----|------|
   |example-cpp-minimal-build-static|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-example-cpp-minimal-build-static)](https://github.com/ursacomputing/crossbow/actions/runs/14991686168/job/42116304546)|
   |example-cpp-minimal-build-static-system-dependency|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-example-cpp-minimal-build-static-system-dependency)](https://github.com/ursacomputing/crossbow/actions/runs/14991686644/job/42116307598)|
   |example-cpp-tutorial|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-example-cpp-tutorial)](https://github.com/ursacomputing/crossbow/actions/runs/14991686648/job/42116307022)|
   |test-alpine-linux-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-alpine-linux-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/14991686725/job/42116308247)|
   |test-build-cpp-fuzz|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-build-cpp-fuzz)](https://github.com/ursacomputing/crossbow/actions/runs/14991686372/job/42116306005)|
   |test-conda-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-conda-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/14991686828/job/42116309016)|
   |test-conda-cpp-meson|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-conda-cpp-meson)](https://github.com/ursacomputing/crossbow/actions/runs/14991686765/job/42116308315)|
   |test-conda-cpp-valgrind|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-conda-cpp-valgrind)](https://github.com/ursacomputing/crossbow/actions/runs/14991686603/job/42116307723)|
   |test-cuda-cpp-ubuntu-22.04-cuda-11.7.1|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-cuda-cpp-ubuntu-22.04-cuda-11.7.1)](https://github.com/ursacomputing/crossbow/actions/runs/14991686650/job/42116307039)|
   |test-debian-12-cpp-amd64|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-debian-12-cpp-amd64)](https://github.com/ursacomputing/crossbow/actions/runs/14991686373/job/42116306380)|
   |test-debian-12-cpp-i386|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-debian-12-cpp-i386)](https://github.com/ursacomputing/crossbow/actions/runs/14991686371/job/42116305955)|
   |test-fedora-39-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-fedora-39-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/14991686218/job/42116304534)|
   |test-ubuntu-22.04-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-ubuntu-22.04-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/14991686568/job/42116306822)|
   |test-ubuntu-22.04-cpp-20|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-ubuntu-22.04-cpp-20)](https://github.com/ursacomputing/crossbow/actions/runs/14991686121/job/42116304482)|
   |test-ubuntu-22.04-cpp-bundled|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-ubuntu-22.04-cpp-bundled)](https://github.com/ursacomputing/crossbow/actions/runs/14991686692/job/42116308184)|
   |test-ubuntu-22.04-cpp-emscripten|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-ubuntu-22.04-cpp-emscripten)](https://github.com/ursacomputing/crossbow/actions/runs/14991686124/job/42116304547)|
   |test-ubuntu-22.04-cpp-no-threading|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-ubuntu-22.04-cpp-no-threading)](https://github.com/ursacomputing/crossbow/actions/runs/14991686505/job/42116306753)|
   |test-ubuntu-24.04-cpp|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-ubuntu-24.04-cpp)](https://github.com/ursacomputing/crossbow/actions/runs/14991686382/job/42116306388)|
   |test-ubuntu-24.04-cpp-bundled-offline|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-ubuntu-24.04-cpp-bundled-offline)](https://github.com/ursacomputing/crossbow/actions/runs/14991686366/job/42116305974)|
   |test-ubuntu-24.04-cpp-gcc-13-bundled|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-ubuntu-24.04-cpp-gcc-13-bundled)](https://github.com/ursacomputing/crossbow/actions/runs/14991686406/job/42116306401)|
   |test-ubuntu-24.04-cpp-gcc-14|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-ubuntu-24.04-cpp-gcc-14)](https://github.com/ursacomputing/crossbow/actions/runs/14991686727/job/42116308673)|
   |test-ubuntu-24.04-cpp-minimal-with-formats|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-ubuntu-24.04-cpp-minimal-with-formats)](https://github.com/ursacomputing/crossbow/actions/runs/14991686203/job/42116304963)|
   |test-ubuntu-24.04-cpp-thread-sanitizer|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-b8b690defb-github-test-ubuntu-24.04-cpp-thread-sanitizer)](https://github.com/ursacomputing/crossbow/actions/runs/14991686464/job/42116306470)|


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086123629


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,412 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+namespace {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_level parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_level Normalization level (default 0)
+// @return The mask used to compare against the rolling hash
+uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size, int norm_level) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be non-negative");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes; also divide by the number of gearhash tables to have a
+  // a more gaussian-like distribution
+  int64_t target_size = (avg_chunk_size - min_chunk_size) / kNumGearhashTables;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // a user defined `norm_level` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_level we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_level is 0 by
+  // default
+  int effective_bits = mask_bits - norm_level;
+
+  if (effective_bits < 1 || effective_bits > 63) {
+    throw ParquetException(
+        "The number of bits in the CDC mask must be between 1 and 63, got " +
+        std::to_string(effective_bits));
+  } else {
+    // create the mask by setting the top bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+}  // namespace
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int norm_level)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_level)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the

Review Comment:
   Sounds good, I'm adding it to the implementation then.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2086666987


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1689 @@
+// Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   Do we have an existing setup to detect sanitizer/valgrind builds?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on PR #45360:
URL: https://github.com/apache/arrow/pull/45360#issuecomment-2876873324

   I collected the possible follow-ups:
   - [C++][Parquet] Support CDC in `WriteBatch` and `WriteBatchSpaced` https://github.com/apache/arrow/pull/45360#discussion_r1978502781
   - [C++][Parquet] Center the chunk size around `avg_chunk_size` if `norm_level` is not `0` https://github.com/apache/arrow/pull/45360#discussion_r1989656894
   - [C++][Parquet] Consider enforcing little-endian intepretation during chunking https://github.com/apache/arrow/pull/45360#discussion_r2024843403
   - [C++][Parquet] Avoid unconditional array slice in `WriteArrow`, `WriteArrowDense` and `WriteArrowDictionary` if `value_offset == 0` https://github.com/apache/arrow/pull/45360#discussion_r2082990349
   - [C++][Python][Parquet] Store the properties used to write the parquet file as arbitrary metadata https://github.com/apache/arrow/pull/45360#discussion_r2083706032
   - [Python][Parquet] Implement a hierarchy of `Parquet` exceptions rather than raising `OSError` https://github.com/apache/arrow/pull/45360#discussion_r2086152438


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["wgtmac (via GitHub)" <gi...@apache.org>]

wgtmac commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2083708018


##########
cpp/src/parquet/chunker_internal.h:
##########
@@ -0,0 +1,144 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <vector>
+
+#include "arrow/array.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+// Represents a chunk of data with level offsets and value offsets due to the
+// record shredding for nested data.
+struct Chunk {
+  // The start offset of this chunk inside the given levels
+  int64_t level_offset;
+  // The start offset of this chunk inside the given values array
+  int64_t value_offset;
+  // The length of the chunk in levels
+  int64_t levels_to_write;
+};
+
+/// CDC (Content-Defined Chunking) is a technique that divides data into variable-sized
+/// chunks based on the content of the data itself, rather than using fixed-size
+/// boundaries.
+///
+/// For example, given this sequence of values in a column:
+///
+/// File1:    [1,2,3,   4,5,6,   7,8,9]
+///            chunk1   chunk2   chunk3
+///
+/// Assume there is an inserted value between 3 and 4:
+///
+/// File2:     [1,2,3,0,  4,5,6,   7,8,9]
+///            new-chunk  chunk2   chunk3
+///
+/// The chunking process will adjust to maintain stable boundaries across data
+/// modifications. Each chunk defines a new parquet data page which is contiguously
+/// written out to the file. Since each page compressed independently, the files' contents
+/// would look like the following with unique page identifiers:
+///
+/// File1:     [Page1][Page2][Page3]...
+/// File2:     [Page4][Page2][Page3]...

Review Comment:
   I just don't quite understand how does the rolling hash can perfectly produce page1, page4 as above. I need to read the paper and blogs more carefully but cannot promise that my math background allows me to totally understand it. :)



##########
cpp/src/parquet/properties.h:
##########
@@ -275,10 +307,38 @@ class PARQUET_EXPORT WriterProperties {
           page_checksum_enabled_(properties.page_checksum_enabled()),
           size_statistics_level_(properties.size_statistics_level()),
           sorting_columns_(properties.sorting_columns()),
-          default_column_properties_(properties.default_column_properties()) {}
+          default_column_properties_(properties.default_column_properties()),
+          content_defined_chunking_enabled_(
+              properties.content_defined_chunking_enabled()),
+          content_defined_chunking_options_(
+              properties.content_defined_chunking_options()) {}
 
     virtual ~Builder() {}
 
+    /// \brief EXPERIMENTAL: Use content-defined page chunking for all columns.
+    ///
+    /// Optimize parquet files for content addressable storage (CAS) systems by writing

Review Comment:
   It is worth noting that only WriteArrow interface is supported.



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,412 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+namespace {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_level parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_level Normalization level (default 0)
+// @return The mask used to compare against the rolling hash
+uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size, int norm_level) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be non-negative");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes; also divide by the number of gearhash tables to have a
+  // a more gaussian-like distribution
+  int64_t target_size = (avg_chunk_size - min_chunk_size) / kNumGearhashTables;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // a user defined `norm_level` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_level we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_level is 0 by
+  // default
+  int effective_bits = mask_bits - norm_level;
+
+  if (effective_bits < 1 || effective_bits > 63) {
+    throw ParquetException(
+        "The number of bits in the CDC mask must be between 1 and 63, got " +
+        std::to_string(effective_bits));
+  } else {
+    // create the mask by setting the top bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+}  // namespace
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int norm_level)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_level)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+    has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+  }
+
+  template <int kByteWidth>
+  void Roll(const uint8_t* value) {
+    // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+    // true if the hash matches the mask.
+    chunk_size_ += kByteWidth;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (size_t i = 0; i < kByteWidth; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  template <typename T>
+  void Roll(const T* value) {
+    return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+  }
+
+  void Roll(const uint8_t* value, int64_t length) {
+    // Update the rolling hash with a binary-like value, set has_matched_ to true if the
+    // hash matches the mask.
+    chunk_size_ += length;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (auto i = 0; i < length; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  bool NeedNewChunk() {
+    // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+    // set to true if we encountered a match since the last NeedNewChunk() call
+    if (ARROW_PREDICT_FALSE(has_matched_)) {
+      has_matched_ = false;
+      // in order to have a normal distribution of chunk sizes, we only create a new chunk
+      // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+      // different gearhash table (gearhash's chunk size has geometric distribution, and
+      // we use central limit theorem to approximate normal distribution, see
+      // section 6.2.1 in paper https://www.cidrdb.org/cidr2023/papers/p43-low.pdf)
+      if (ARROW_PREDICT_FALSE(++nth_run_ >= kNumGearhashTables)) {
+        // note that we choose not to reset the rolling hash state here, nor anywhere else
+        // in the code, in practice this doesn't seem to affect the chunking effectiveness
+        nth_run_ = 0;
+        chunk_size_ = 0;
+        return true;
+      }
+    }
+    if (ARROW_PREDICT_FALSE(chunk_size_ >= max_chunk_size_)) {
+      // we have a hard limit on the maximum chunk size, note that we don't reset the
+      // rolling hash state here, so the next NeedNewChunk() call will continue from the
+      // current state
+      chunk_size_ = 0;
+      return true;
+    }
+    return false;
+  }
+
+  void ValidateChunks(const std::vector<Chunk>& chunks, int64_t num_levels) const {
+    // chunks must be non-empty and monotonic increasing
+    ARROW_DCHECK(!chunks.empty());
+
+    // the first chunk must start at the first level
+    auto first_chunk = chunks.front();
+    ARROW_DCHECK_EQ(first_chunk.level_offset, 0);
+    ARROW_DCHECK_EQ(first_chunk.value_offset, 0);
+
+    // the following chunks must be contiguous, non-overlapping and monotonically
+    // increasing
+    auto sum_levels = first_chunk.levels_to_write;
+    for (size_t i = 1; i < chunks.size(); ++i) {
+      auto chunk = chunks[i];
+      auto prev_chunk = chunks[i - 1];
+      ARROW_DCHECK_GT(chunk.levels_to_write, 0);
+      ARROW_DCHECK_GE(chunk.value_offset, prev_chunk.value_offset);
+      ARROW_DCHECK_EQ(chunk.level_offset,
+                      prev_chunk.level_offset + prev_chunk.levels_to_write);
+      sum_levels += chunk.levels_to_write;
+    }
+    ARROW_DCHECK_EQ(sum_levels, num_levels);
+
+    // the last chunk must end at the last level
+    auto last_chunk = chunks.back();
+    ARROW_DCHECK_EQ(last_chunk.level_offset + last_chunk.levels_to_write, num_levels);
+  }
+
+  template <typename RollFunc>
+  std::vector<Chunk> Calculate(const int16_t* def_levels, const int16_t* rep_levels,
+                               int64_t num_levels, const RollFunc& RollValue) {
+    // Calculate the chunk boundaries for typed Arrow arrays.
+    std::vector<Chunk> chunks;
+    int64_t offset;
+    int64_t prev_offset = 0;
+    int64_t prev_value_offset = 0;
+    bool has_def_levels = level_info_.def_level > 0;
+    bool has_rep_levels = level_info_.rep_level > 0;
+
+    if (!has_rep_levels && !has_def_levels) {
+      // fastest path for non-nested non-null data
+      for (offset = 0; offset < num_levels; ++offset) {
+        RollValue(offset);
+        if (NeedNewChunk()) {
+          chunks.push_back({prev_offset, prev_offset, offset - prev_offset});
+          prev_offset = offset;
+        }
+      }
+      // set the previous value offset to add the last chunk
+      prev_value_offset = prev_offset;
+    } else if (!has_rep_levels) {
+      // non-nested data with nulls
+      int16_t def_level;
+      for (int64_t offset = 0; offset < num_levels; ++offset) {
+        def_level = def_levels[offset];
+
+        Roll(&def_level);
+        if (def_level == level_info_.def_level) {
+          RollValue(offset);
+        }
+        if (NeedNewChunk()) {
+          chunks.push_back({prev_offset, prev_offset, offset - prev_offset});
+          prev_offset = offset;
+        }
+      }
+      // set the previous value offset to add the last chunk
+      prev_value_offset = prev_offset;
+    } else {
+      // nested data with nulls
+      int16_t def_level;
+      int16_t rep_level;
+      int64_t value_offset = 0;
+
+      for (offset = 0; offset < num_levels; ++offset) {
+        def_level = def_levels[offset];
+        rep_level = rep_levels[offset];
+
+        Roll(&def_level);
+        Roll(&rep_level);
+        if (def_level == level_info_.def_level) {
+          RollValue(value_offset);
+        }
+
+        if ((rep_level == 0) && NeedNewChunk()) {

Review Comment:
   ```suggestion
           if (rep_level == 0 && NeedNewChunk()) {
   ```



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,412 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+namespace {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_level parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_level Normalization level (default 0)
+// @return The mask used to compare against the rolling hash
+uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size, int norm_level) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be non-negative");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes; also divide by the number of gearhash tables to have a
+  // a more gaussian-like distribution
+  int64_t target_size = (avg_chunk_size - min_chunk_size) / kNumGearhashTables;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // a user defined `norm_level` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_level we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_level is 0 by
+  // default
+  int effective_bits = mask_bits - norm_level;
+
+  if (effective_bits < 1 || effective_bits > 63) {
+    throw ParquetException(
+        "The number of bits in the CDC mask must be between 1 and 63, got " +
+        std::to_string(effective_bits));
+  } else {
+    // create the mask by setting the top bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+}  // namespace
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int norm_level)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_level)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+    has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+  }
+
+  template <int kByteWidth>
+  void Roll(const uint8_t* value) {
+    // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+    // true if the hash matches the mask.
+    chunk_size_ += kByteWidth;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (size_t i = 0; i < kByteWidth; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  template <typename T>
+  void Roll(const T* value) {
+    return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+  }
+
+  void Roll(const uint8_t* value, int64_t length) {
+    // Update the rolling hash with a binary-like value, set has_matched_ to true if the
+    // hash matches the mask.
+    chunk_size_ += length;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (auto i = 0; i < length; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  bool NeedNewChunk() {
+    // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+    // set to true if we encountered a match since the last NeedNewChunk() call
+    if (ARROW_PREDICT_FALSE(has_matched_)) {
+      has_matched_ = false;
+      // in order to have a normal distribution of chunk sizes, we only create a new chunk
+      // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+      // different gearhash table (gearhash's chunk size has geometric distribution, and
+      // we use central limit theorem to approximate normal distribution, see
+      // section 6.2.1 in paper https://www.cidrdb.org/cidr2023/papers/p43-low.pdf)
+      if (ARROW_PREDICT_FALSE(++nth_run_ >= kNumGearhashTables)) {
+        // note that we choose not to reset the rolling hash state here, nor anywhere else
+        // in the code, in practice this doesn't seem to affect the chunking effectiveness
+        nth_run_ = 0;
+        chunk_size_ = 0;
+        return true;
+      }
+    }
+    if (ARROW_PREDICT_FALSE(chunk_size_ >= max_chunk_size_)) {
+      // we have a hard limit on the maximum chunk size, note that we don't reset the
+      // rolling hash state here, so the next NeedNewChunk() call will continue from the
+      // current state
+      chunk_size_ = 0;
+      return true;
+    }
+    return false;
+  }
+
+  void ValidateChunks(const std::vector<Chunk>& chunks, int64_t num_levels) const {
+    // chunks must be non-empty and monotonic increasing
+    ARROW_DCHECK(!chunks.empty());
+
+    // the first chunk must start at the first level
+    auto first_chunk = chunks.front();
+    ARROW_DCHECK_EQ(first_chunk.level_offset, 0);
+    ARROW_DCHECK_EQ(first_chunk.value_offset, 0);
+
+    // the following chunks must be contiguous, non-overlapping and monotonically
+    // increasing
+    auto sum_levels = first_chunk.levels_to_write;
+    for (size_t i = 1; i < chunks.size(); ++i) {
+      auto chunk = chunks[i];
+      auto prev_chunk = chunks[i - 1];
+      ARROW_DCHECK_GT(chunk.levels_to_write, 0);
+      ARROW_DCHECK_GE(chunk.value_offset, prev_chunk.value_offset);
+      ARROW_DCHECK_EQ(chunk.level_offset,
+                      prev_chunk.level_offset + prev_chunk.levels_to_write);
+      sum_levels += chunk.levels_to_write;
+    }
+    ARROW_DCHECK_EQ(sum_levels, num_levels);
+
+    // the last chunk must end at the last level
+    auto last_chunk = chunks.back();
+    ARROW_DCHECK_EQ(last_chunk.level_offset + last_chunk.levels_to_write, num_levels);
+  }
+
+  template <typename RollFunc>
+  std::vector<Chunk> Calculate(const int16_t* def_levels, const int16_t* rep_levels,
+                               int64_t num_levels, const RollFunc& RollValue) {
+    // Calculate the chunk boundaries for typed Arrow arrays.
+    std::vector<Chunk> chunks;
+    int64_t offset;
+    int64_t prev_offset = 0;
+    int64_t prev_value_offset = 0;
+    bool has_def_levels = level_info_.def_level > 0;
+    bool has_rep_levels = level_info_.rep_level > 0;
+
+    if (!has_rep_levels && !has_def_levels) {
+      // fastest path for non-nested non-null data
+      for (offset = 0; offset < num_levels; ++offset) {
+        RollValue(offset);
+        if (NeedNewChunk()) {
+          chunks.push_back({prev_offset, prev_offset, offset - prev_offset});
+          prev_offset = offset;
+        }
+      }
+      // set the previous value offset to add the last chunk
+      prev_value_offset = prev_offset;
+    } else if (!has_rep_levels) {
+      // non-nested data with nulls
+      int16_t def_level;
+      for (int64_t offset = 0; offset < num_levels; ++offset) {
+        def_level = def_levels[offset];
+
+        Roll(&def_level);
+        if (def_level == level_info_.def_level) {
+          RollValue(offset);
+        }
+        if (NeedNewChunk()) {
+          chunks.push_back({prev_offset, prev_offset, offset - prev_offset});
+          prev_offset = offset;
+        }
+      }
+      // set the previous value offset to add the last chunk
+      prev_value_offset = prev_offset;
+    } else {
+      // nested data with nulls
+      int16_t def_level;
+      int16_t rep_level;
+      int64_t value_offset = 0;
+
+      for (offset = 0; offset < num_levels; ++offset) {
+        def_level = def_levels[offset];
+        rep_level = rep_levels[offset];
+
+        Roll(&def_level);
+        Roll(&rep_level);
+        if (def_level == level_info_.def_level) {
+          RollValue(value_offset);
+        }
+
+        if ((rep_level == 0) && NeedNewChunk()) {
+          // if we are at a record boundary and need a new chunk, we create a new chunk
+          auto levels_to_write = offset - prev_offset;
+          if (levels_to_write > 0) {
+            chunks.push_back({prev_offset, prev_value_offset, levels_to_write});
+            prev_offset = offset;
+            prev_value_offset = value_offset;
+          }
+        }
+        if (def_level >= level_info_.repeated_ancestor_def_level) {
+          // we only increment the value offset if we have a leaf value
+          ++value_offset;
+        }
+      }
+    }
+
+    // add the last chunk if we have any levels left
+    if (prev_offset < num_levels) {
+      chunks.push_back({prev_offset, prev_value_offset, num_levels - prev_offset});
+    }
+#ifndef NDEBUG
+    ValidateChunks(chunks, num_levels);
+#endif
+
+    return chunks;
+  }
+
+  template <int kByteWidth>
+  std::vector<Chunk> CalculateFixedWidth(const int16_t* def_levels,
+                                         const int16_t* rep_levels, int64_t num_levels,
+                                         const ::arrow::Array& values) {
+    const uint8_t* raw_values =
+        values.data()->GetValues<uint8_t>(1, 0) + values.offset() * kByteWidth;

Review Comment:
   ```suggestion
           values.data()->GetValues<uint8_t>(/*i=*/1, /*absolute_offset=*/0) + values.offset() * kByteWidth;
   ```
   
   Just for better readability.



##########
python/pyarrow/_parquet.pxd:
##########
@@ -495,6 +500,9 @@ cdef extern from "parquet/api/writer.h" namespace "parquet" nogil:
             Builder* disable_write_page_index()
             Builder* enable_page_checksum()
             Builder* disable_page_checksum()
+            Builder* enable_content_defined_chunking()
+            Builder* disable_content_defined_chunking()
+            Builder* content_defined_chunking_options(const CdcOptions options)

Review Comment:
   ```suggestion
               Builder* content_defined_chunking_options(CdcOptions options)
   ```
   
   It seems that we can remove `const`?



##########
cpp/src/parquet/properties.h:
##########
@@ -260,7 +290,9 @@ class PARQUET_EXPORT WriterProperties {
           created_by_(DEFAULT_CREATED_BY),
           store_decimal_as_integer_(false),
           page_checksum_enabled_(false),
-          size_statistics_level_(DEFAULT_SIZE_STATISTICS_LEVEL) {}
+          size_statistics_level_(DEFAULT_SIZE_STATISTICS_LEVEL),
+          content_defined_chunking_enabled_(false),
+          content_defined_chunking_options_(kDefaultCdcOptions) {}

Review Comment:
   Do we still need `kDefaultCdcOptions`? `content_defined_chunking_options_({})` seems to be enough.



##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1687 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include <gtest/gtest.h>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::internal::checked_cast;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = checked_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = checked_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size =
+          checked_cast<const ::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = checked_cast<const ::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array, GenerateArray(struct_type->field(i),
+                                                              length, seed + i * 10));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      // Repeat the same pattern in the list array:
+      // null, empty list, list of 1 element, list of 3 elements
+      if (length % 4 != 0) {
+        return Status::Invalid(
+            "Length must be divisible by 4 when generating list arrays, but got: ",
+            length);
+      }
+      auto values_array_length = length * 4;
+      auto list_type = checked_cast<const ::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array,
+                            GenerateArray(value_field, values_array_length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      RETURN_NOT_OK(offset_builder.Reserve(length + 1));
+      RETURN_NOT_OK(bitmap_builder.Reserve(length));
+
+      int32_t num_nulls = 0;
+      RETURN_NOT_OK(offset_builder.Append(0));
+      for (auto offset = 0; offset < length; offset += 4) {
+        if (nullable) {
+          // add a null
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          RETURN_NOT_OK(offset_builder.Append(offset));
+          num_nulls += 1;
+        } else {
+          // add an empty list
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          RETURN_NOT_OK(offset_builder.Append(offset));
+        }
+        // add an empty list
+        RETURN_NOT_OK(bitmap_builder.Append(true));
+        RETURN_NOT_OK(offset_builder.Append(offset));
+        // add a list of 1 element
+        RETURN_NOT_OK(bitmap_builder.Append(true));
+        RETURN_NOT_OK(offset_builder.Append(offset + 1));
+        // add a list of 3 elements
+        RETURN_NOT_OK(bitmap_builder.Append(true));
+        RETURN_NOT_OK(offset_builder.Append(offset + 4));
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = checked_cast<const ::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+TEST(TestGenerateArray, Integer) {
+  auto field = ::arrow::field("a", ::arrow::int32());
+  ASSERT_OK_AND_ASSIGN(auto array, GenerateArray(field, /*length=*/10, /*seed=*/0));
+  ASSERT_OK(array->ValidateFull());
+  ASSERT_EQ(array->length(), 10);
+  ASSERT_TRUE(array->type()->Equals(::arrow::int32()));
+  ASSERT_EQ(array->null_count(), 1);
+}
+
+TEST(TestGenerateArray, ListOfInteger) {
+  auto field = ::arrow::field("a", ::arrow::list(::arrow::int32()));
+  auto length = 12;
+  ASSERT_OK_AND_ASSIGN(auto array, GenerateArray(field, length, /*seed=*/0));
+  ASSERT_OK(array->ValidateFull());
+  ASSERT_EQ(array->length(), length);
+
+  for (size_t i = 0; i < 12; i += 4) {
+    // Assert the first element is null
+    ASSERT_TRUE(array->IsNull(i));
+
+    // Assert the second element is an empty list
+    ASSERT_TRUE(array->IsValid(i + 1));
+    auto list_array = std::static_pointer_cast<::arrow::ListArray>(array);
+    ASSERT_EQ(list_array->value_length(i + 1), 0);
+
+    // Assert the third element has length 1
+    ASSERT_TRUE(array->IsValid(i + 2));
+    ASSERT_EQ(list_array->value_length(i + 2), 1);
+
+    // Assert the fourth element has length 3
+    ASSERT_TRUE(array->IsValid(i + 3));
+    ASSERT_EQ(list_array->value_length(i + 3), 3);
+  }
+
+  ASSERT_NOT_OK(GenerateArray(field, 3, /*seed=*/0));
+  ASSERT_OK(GenerateArray(field, 8, /*seed=*/0));
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, ++seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  auto props = default_arrow_reader_properties();
+
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(props)
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    int64_t row_group_length = 1024 * 1024, bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      {min_chunk_size, max_chunk_size, /*norm_level=*/0});
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_length,
+                           write_props, arrow_props));
+  ARROW_ASSIGN_OR_RAISE(auto buffer, sink->Finish());
+
+  // check whether the schema has extension types, if not we can easily ensure that
+  // the parquet seralization is roundtripable with CDC enabled
+  bool validate_roundtrip = true;
+  for (const auto& field : table->schema()->fields()) {
+    if (field->type()->id() == ::arrow::Type::EXTENSION) {
+      validate_roundtrip = false;

Review Comment:
   Why extension type does not support roundtrip validation? What if the arrow schema has been preserved in the parquet metadata?



##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();
+    auto min_length = ElementCount(min_chunk_size, byte_width, nullable);
+    auto max_length = ElementCount(max_chunk_size, byte_width, nullable);
+    AssertAllBetween(base_info.page_lengths, min_length, max_length);
+    AssertAllBetween(modified_info.page_lengths, min_length, max_length);
+  } else if (::arrow::is_base_binary_like(dtype->id()) && !enable_dictionary) {
+    AssertAllBetween(base_info.page_sizes, min_chunk_size, max_chunk_size);
+    AssertAllBetween(modified_info.page_sizes, min_chunk_size, max_chunk_size);
+  }
+}
+
+constexpr int64_t kMinChunkSize = 8 * 1024;
+constexpr int64_t kMaxChunkSize = 32 * 1024;
+constexpr int64_t kPartSize = 128 * 1024;
+constexpr int64_t kEditSize = 128;
+
+struct CaseConfig {
+  // Arrow data type to generate the testing data for
+  std::shared_ptr<::arrow::DataType> dtype;
+  // Whether the data type is nullable
+  bool is_nullable;
+  // Approximate number of bytes per record to calculate the number of elements to
+  // generate
+  size_t bytes_per_record;
+  // Data page version to use
+  ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1;
+};
+
+// Define PrintTo for MyStruct
+void PrintTo(const CaseConfig& param, std::ostream* os) {
+  *os << "{ " << param.dtype->ToString();
+  if (param.is_nullable) {
+    *os << " nullable";
+  }
+  *os << " }";
+}
+
+class TestCDC : public ::testing::Test {
+ public:
+  uint64_t GetMask(const ContentDefinedChunker& cdc) const { return cdc.GetMask(); }
+};
+
+TEST_F(TestCDC, RollingHashMaskCalculation) {
+  auto le = LevelInfo();
+  auto min_size = 256 * 1024;
+  auto max_size = 1024 * 1024;
+
+  auto cdc0 = ContentDefinedChunker(le, min_size, max_size, 0);
+  ASSERT_EQ(GetMask(cdc0), 0xFFFE000000000000);
+
+  auto cdc1 = ContentDefinedChunker(le, min_size, max_size, 1);
+  ASSERT_EQ(GetMask(cdc1), 0xFFFC000000000000);
+
+  auto cdc2 = ContentDefinedChunker(le, min_size, max_size, 2);
+  ASSERT_EQ(GetMask(cdc2), 0xFFF8000000000000);
+
+  auto cdc3 = ContentDefinedChunker(le, min_size, max_size, 3);
+  ASSERT_EQ(GetMask(cdc3), 0xFFF0000000000000);
+
+  auto cdc4 = ContentDefinedChunker(le, min_size, max_size, -1);
+  ASSERT_EQ(GetMask(cdc4), 0xFFFF000000000000);
+
+  // this is the smallest possible mask always matching, by using 8 hashtables
+  // we are going to have a match every 8 bytes; this is an unrealistic case
+  // but checking for the correctness of the mask calculation
+  auto cdc5 = ContentDefinedChunker(le, 0, 16, 0);
+  ASSERT_EQ(GetMask(cdc5), 0x0000000000000000);
+
+  auto cdc6 = ContentDefinedChunker(le, 0, 32, 1);
+  ASSERT_EQ(GetMask(cdc6), 0x0000000000000000);
+
+  auto cdc7 = ContentDefinedChunker(le, 0, 16, -1);
+  ASSERT_EQ(GetMask(cdc7), 0x8000000000000000);
+
+  // another unrealistic case, checking for the validation
+  auto cdc8 = ContentDefinedChunker(le, 128, 384, -60);
+  ASSERT_EQ(GetMask(cdc8), 0xFFFFFFFFFFFFFFFF);
+}
+
+TEST_F(TestCDC, WriteSingleColumnParquetFile) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  std::vector<int32_t> numbers = {1, 2, 3, 4, 5};
+  std::vector<uint8_t> valid_bits = {1, 0, 1, 0, 1};
+  EXPECT_THROW(
+      int_column_writer.WriteBatch(numbers.size(), nullptr, nullptr, numbers.data()),
+      ParquetException);

Review Comment:
   Yes we do. You can find `EXPECT_THROW_THAT` from the Parquet subdirectory.



##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,412 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+namespace {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_level parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_level Normalization level (default 0)
+// @return The mask used to compare against the rolling hash
+uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size, int norm_level) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be non-negative");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes; also divide by the number of gearhash tables to have a
+  // a more gaussian-like distribution
+  int64_t target_size = (avg_chunk_size - min_chunk_size) / kNumGearhashTables;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // a user defined `norm_level` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_level we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_level is 0 by
+  // default
+  int effective_bits = mask_bits - norm_level;
+
+  if (effective_bits < 1 || effective_bits > 63) {
+    throw ParquetException(
+        "The number of bits in the CDC mask must be between 1 and 63, got " +
+        std::to_string(effective_bits));
+  } else {
+    // create the mask by setting the top bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+}  // namespace
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int norm_level)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_level)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the

Review Comment:
   The trade-off here is that all writers to the same CAS should apply same `min_chunk_size` config so we can safely ignore the heading bytes before `min_chunk_size`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2085160551


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,412 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+namespace {
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_level parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_level Normalization level (default 0)
+// @return The mask used to compare against the rolling hash
+uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size, int norm_level) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be non-negative");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes; also divide by the number of gearhash tables to have a
+  // a more gaussian-like distribution
+  int64_t target_size = (avg_chunk_size - min_chunk_size) / kNumGearhashTables;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // a user defined `norm_level` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_level we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_level is 0 by
+  // default
+  int effective_bits = mask_bits - norm_level;
+
+  if (effective_bits < 1 || effective_bits > 63) {
+    throw ParquetException(
+        "The number of bits in the CDC mask must be between 1 and 63, got " +
+        std::to_string(effective_bits));
+  } else {
+    // create the mask by setting the top bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+}  // namespace
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int norm_level)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_level)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+    has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+  }
+
+  template <int kByteWidth>
+  void Roll(const uint8_t* value) {
+    // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+    // true if the hash matches the mask.
+    chunk_size_ += kByteWidth;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (size_t i = 0; i < kByteWidth; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  template <typename T>
+  void Roll(const T* value) {
+    return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+  }
+
+  void Roll(const uint8_t* value, int64_t length) {
+    // Update the rolling hash with a binary-like value, set has_matched_ to true if the
+    // hash matches the mask.
+    chunk_size_ += length;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (auto i = 0; i < length; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  bool NeedNewChunk() {
+    // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+    // set to true if we encountered a match since the last NeedNewChunk() call
+    if (ARROW_PREDICT_FALSE(has_matched_)) {
+      has_matched_ = false;
+      // in order to have a normal distribution of chunk sizes, we only create a new chunk
+      // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+      // different gearhash table (gearhash's chunk size has geometric distribution, and
+      // we use central limit theorem to approximate normal distribution, see
+      // section 6.2.1 in paper https://www.cidrdb.org/cidr2023/papers/p43-low.pdf)
+      if (ARROW_PREDICT_FALSE(++nth_run_ >= kNumGearhashTables)) {
+        // note that we choose not to reset the rolling hash state here, nor anywhere else
+        // in the code, in practice this doesn't seem to affect the chunking effectiveness
+        nth_run_ = 0;
+        chunk_size_ = 0;
+        return true;
+      }
+    }
+    if (ARROW_PREDICT_FALSE(chunk_size_ >= max_chunk_size_)) {
+      // we have a hard limit on the maximum chunk size, note that we don't reset the
+      // rolling hash state here, so the next NeedNewChunk() call will continue from the
+      // current state
+      chunk_size_ = 0;
+      return true;
+    }
+    return false;
+  }
+
+  void ValidateChunks(const std::vector<Chunk>& chunks, int64_t num_levels) const {
+    // chunks must be non-empty and monotonic increasing
+    ARROW_DCHECK(!chunks.empty());
+
+    // the first chunk must start at the first level
+    auto first_chunk = chunks.front();
+    ARROW_DCHECK_EQ(first_chunk.level_offset, 0);
+    ARROW_DCHECK_EQ(first_chunk.value_offset, 0);
+
+    // the following chunks must be contiguous, non-overlapping and monotonically
+    // increasing
+    auto sum_levels = first_chunk.levels_to_write;
+    for (size_t i = 1; i < chunks.size(); ++i) {
+      auto chunk = chunks[i];
+      auto prev_chunk = chunks[i - 1];
+      ARROW_DCHECK_GT(chunk.levels_to_write, 0);
+      ARROW_DCHECK_GE(chunk.value_offset, prev_chunk.value_offset);
+      ARROW_DCHECK_EQ(chunk.level_offset,
+                      prev_chunk.level_offset + prev_chunk.levels_to_write);
+      sum_levels += chunk.levels_to_write;
+    }
+    ARROW_DCHECK_EQ(sum_levels, num_levels);
+
+    // the last chunk must end at the last level
+    auto last_chunk = chunks.back();
+    ARROW_DCHECK_EQ(last_chunk.level_offset + last_chunk.levels_to_write, num_levels);
+  }
+
+  template <typename RollFunc>
+  std::vector<Chunk> Calculate(const int16_t* def_levels, const int16_t* rep_levels,
+                               int64_t num_levels, const RollFunc& RollValue) {
+    // Calculate the chunk boundaries for typed Arrow arrays.
+    std::vector<Chunk> chunks;
+    int64_t offset;
+    int64_t prev_offset = 0;
+    int64_t prev_value_offset = 0;
+    bool has_def_levels = level_info_.def_level > 0;
+    bool has_rep_levels = level_info_.rep_level > 0;
+
+    if (!has_rep_levels && !has_def_levels) {
+      // fastest path for non-nested non-null data
+      for (offset = 0; offset < num_levels; ++offset) {
+        RollValue(offset);
+        if (NeedNewChunk()) {
+          chunks.push_back({prev_offset, prev_offset, offset - prev_offset});
+          prev_offset = offset;
+        }
+      }
+      // set the previous value offset to add the last chunk
+      prev_value_offset = prev_offset;
+    } else if (!has_rep_levels) {
+      // non-nested data with nulls
+      int16_t def_level;
+      for (int64_t offset = 0; offset < num_levels; ++offset) {
+        def_level = def_levels[offset];
+
+        Roll(&def_level);
+        if (def_level == level_info_.def_level) {
+          RollValue(offset);
+        }
+        if (NeedNewChunk()) {
+          chunks.push_back({prev_offset, prev_offset, offset - prev_offset});
+          prev_offset = offset;
+        }
+      }
+      // set the previous value offset to add the last chunk
+      prev_value_offset = prev_offset;
+    } else {
+      // nested data with nulls
+      int16_t def_level;
+      int16_t rep_level;
+      int64_t value_offset = 0;
+
+      for (offset = 0; offset < num_levels; ++offset) {
+        def_level = def_levels[offset];
+        rep_level = rep_levels[offset];
+
+        Roll(&def_level);
+        Roll(&rep_level);
+        if (def_level == level_info_.def_level) {
+          RollValue(value_offset);
+        }
+
+        if ((rep_level == 0) && NeedNewChunk()) {

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2085254543


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();
+    auto min_length = ElementCount(min_chunk_size, byte_width, nullable);
+    auto max_length = ElementCount(max_chunk_size, byte_width, nullable);
+    AssertAllBetween(base_info.page_lengths, min_length, max_length);
+    AssertAllBetween(modified_info.page_lengths, min_length, max_length);
+  } else if (::arrow::is_base_binary_like(dtype->id()) && !enable_dictionary) {
+    AssertAllBetween(base_info.page_sizes, min_chunk_size, max_chunk_size);
+    AssertAllBetween(modified_info.page_sizes, min_chunk_size, max_chunk_size);
+  }
+}
+
+constexpr int64_t kMinChunkSize = 8 * 1024;
+constexpr int64_t kMaxChunkSize = 32 * 1024;
+constexpr int64_t kPartSize = 128 * 1024;
+constexpr int64_t kEditSize = 128;
+
+struct CaseConfig {
+  // Arrow data type to generate the testing data for
+  std::shared_ptr<::arrow::DataType> dtype;
+  // Whether the data type is nullable
+  bool is_nullable;
+  // Approximate number of bytes per record to calculate the number of elements to
+  // generate
+  size_t bytes_per_record;
+  // Data page version to use
+  ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1;
+};
+
+// Define PrintTo for MyStruct
+void PrintTo(const CaseConfig& param, std::ostream* os) {
+  *os << "{ " << param.dtype->ToString();
+  if (param.is_nullable) {
+    *os << " nullable";
+  }
+  *os << " }";
+}
+
+class TestCDC : public ::testing::Test {
+ public:
+  uint64_t GetMask(const ContentDefinedChunker& cdc) const { return cdc.GetMask(); }
+};
+
+TEST_F(TestCDC, RollingHashMaskCalculation) {
+  auto le = LevelInfo();
+  auto min_size = 256 * 1024;
+  auto max_size = 1024 * 1024;
+
+  auto cdc0 = ContentDefinedChunker(le, min_size, max_size, 0);
+  ASSERT_EQ(GetMask(cdc0), 0xFFFE000000000000);
+
+  auto cdc1 = ContentDefinedChunker(le, min_size, max_size, 1);
+  ASSERT_EQ(GetMask(cdc1), 0xFFFC000000000000);
+
+  auto cdc2 = ContentDefinedChunker(le, min_size, max_size, 2);
+  ASSERT_EQ(GetMask(cdc2), 0xFFF8000000000000);
+
+  auto cdc3 = ContentDefinedChunker(le, min_size, max_size, 3);
+  ASSERT_EQ(GetMask(cdc3), 0xFFF0000000000000);
+
+  auto cdc4 = ContentDefinedChunker(le, min_size, max_size, -1);
+  ASSERT_EQ(GetMask(cdc4), 0xFFFF000000000000);
+
+  // this is the smallest possible mask always matching, by using 8 hashtables
+  // we are going to have a match every 8 bytes; this is an unrealistic case
+  // but checking for the correctness of the mask calculation
+  auto cdc5 = ContentDefinedChunker(le, 0, 16, 0);
+  ASSERT_EQ(GetMask(cdc5), 0x0000000000000000);
+
+  auto cdc6 = ContentDefinedChunker(le, 0, 32, 1);
+  ASSERT_EQ(GetMask(cdc6), 0x0000000000000000);
+
+  auto cdc7 = ContentDefinedChunker(le, 0, 16, -1);
+  ASSERT_EQ(GetMask(cdc7), 0x8000000000000000);
+
+  // another unrealistic case, checking for the validation
+  auto cdc8 = ContentDefinedChunker(le, 128, 384, -60);
+  ASSERT_EQ(GetMask(cdc8), 0xFFFFFFFFFFFFFFFF);
+}
+
+TEST_F(TestCDC, WriteSingleColumnParquetFile) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  std::vector<int32_t> numbers = {1, 2, 3, 4, 5};
+  std::vector<uint8_t> valid_bits = {1, 0, 1, 0, 1};
+  EXPECT_THROW(
+      int_column_writer.WriteBatch(numbers.size(), nullptr, nullptr, numbers.data()),
+      ParquetException);

Review Comment:
   Thanks, updated to assert on the message now. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2083089886


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,409 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <iterator>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging_internal.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+static_assert(std::size(kGearhashTable) == kNumGearhashTables,
+              "should update CDC code to reflect number of generated hash tables");
+static_assert(sizeof(kGearhashTable) == kNumGearhashTables * 256 * 8,
+              "each table should have 256 entries of 64 bit values");
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes; also divide by the number of gearhash tables to have a
+  // a more gaussian-like distribution
+  int64_t target_size = (avg_chunk_size - min_chunk_size) / kNumGearhashTables;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  int effective_bits = mask_bits - norm_factor;
+
+  if (effective_bits < 1 || effective_bits > 63) {
+    throw ParquetException(
+        "The number of bits in the CDC mask must be between 1 and 63, got " +
+        std::to_string(effective_bits));
+  } else {
+    // create the mask by setting the top bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int norm_factor)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_factor)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(const bool value) {

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025155854


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int8_t norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes
+  int64_t target_size = avg_chunk_size - min_chunk_size;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // 3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  if (norm_factor < -3 || norm_factor > 3) {
+    ARROW_LOG(WARNING) << "norm_factor=" << norm_factor
+                       << " is outside the recommended range [-3, 3]";
+  }
+
+  int mask_adjustement = 3 + norm_factor;
+  int effective_bits = mask_bits - mask_adjustement;
+  if (effective_bits == 0) {
+    return 0;
+  } else if (effective_bits > 64) {
+    throw ParquetException("The number of bits in the mask cannot exceed 64, got " +
+                           std::to_string(effective_bits));
+  } else if (effective_bits < 0) {
+    throw ParquetException(
+        "The difference between min_chunk_size=" + std::to_string(min_chunk_size) +
+        " and max_chunk_size=" + std::to_string(max_chunk_size) +
+        " is too small for the given norm_factor=" + std::to_string(norm_factor) +
+        ", either increase the size range or decrease the norm_factor.");
+  } else {
+    // create the mask by setting the top mask_bits bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int8_t norm_factor)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_factor)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(const bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+    has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+  }
+
+  template <int ByteWidth>
+  void Roll(const uint8_t* value) {
+    // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+    // true if the hash matches the mask.
+
+    chunk_size_ += ByteWidth;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (size_t i = 0; i < ByteWidth; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  template <typename T>
+  void Roll(const T* value) {
+    return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+  }
+
+  void Roll(const uint8_t* value, int64_t length) {
+    // Update the rolling hash with a binary-like value, set has_matched_ to true if the
+    // hash matches the mask.
+
+    chunk_size_ += length;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (auto i = 0; i < length; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  bool NeedNewChunk() {
+    // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+    // set to true if we encountered a match since the last NeedNewChunk() call
+    if (ARROW_PREDICT_FALSE(has_matched_)) {
+      has_matched_ = false;
+      // in order to have a normal distribution of chunk sizes, we only create a new chunk
+      // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+      // different gearhash table (gearhash's chunk size has geometric distribution, and
+      // we use central limit theorem to approximate normal distribution, see
+      // section 6.2.1 in paper https://www.cidrdb.org/cidr2023/papers/p43-low.pdf)
+      if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+        nth_run_ = 0;
+        chunk_size_ = 0;
+        return true;
+      }
+    }
+    if (ARROW_PREDICT_FALSE(chunk_size_ >= max_chunk_size_)) {
+      // we have a hard limit on the maximum chunk size, note that we don't reset the
+      // rolling hash state here, so the next NeedNewChunk() call will continue from the
+      // current state
+      chunk_size_ = 0;
+      return true;
+    }
+    return false;
+  }
+
+  void ValidateChunks(const std::vector<Chunk>& chunks, int64_t num_levels) {
+    // chunks must be non-empty and monotonic increasing
+    DCHECK(!chunks.empty());
+
+    // the first chunk must start at the first level
+    auto first_chunk = chunks.front();
+    DCHECK_EQ(first_chunk.level_offset, 0);
+    DCHECK_EQ(first_chunk.value_offset, 0);
+
+    // the following chunks must be contiguous, non-overlapping and monotonically
+    // increasing
+    auto sum_levels = first_chunk.levels_to_write;
+    for (size_t i = 1; i < chunks.size(); ++i) {
+      auto chunk = chunks[i];
+      auto prev_chunk = chunks[i - 1];
+      DCHECK_GT(chunk.levels_to_write, 0);
+      DCHECK_GE(chunk.value_offset, prev_chunk.value_offset);
+      DCHECK_EQ(chunk.level_offset, prev_chunk.level_offset + prev_chunk.levels_to_write);
+      sum_levels += chunk.levels_to_write;
+    }
+    DCHECK_EQ(sum_levels, num_levels);
+
+    // the last chunk must end at the last level
+    auto last_chunk = chunks.back();
+    DCHECK_EQ(last_chunk.level_offset + last_chunk.levels_to_write, num_levels);
+  }
+
+  template <typename RollFunc>
+  std::vector<Chunk> Calculate(const int16_t* def_levels, const int16_t* rep_levels,
+                               int64_t num_levels, const RollFunc& RollValue) {
+    // Calculate the chunk boundaries for typed Arrow arrays.
+    std::vector<Chunk> chunks;
+    int64_t offset;
+    int64_t prev_offset = 0;
+    int64_t prev_value_offset = 0;
+    bool has_def_levels = level_info_.def_level > 0;
+    bool has_rep_levels = level_info_.rep_level > 0;
+
+    if (!has_rep_levels && !has_def_levels) {
+      // fastest path for non-nested non-null data
+      for (offset = 0; offset < num_levels; ++offset) {
+        RollValue(offset);
+        if (NeedNewChunk()) {
+          chunks.push_back({prev_offset, prev_offset, offset - prev_offset});
+          prev_offset = offset;
+        }
+      }
+      // set the previous value offset to add the last chunk
+      prev_value_offset = prev_offset;
+    } else if (!has_rep_levels) {
+      // non-nested data with nulls
+      int16_t def_level;
+      for (int64_t offset = 0; offset < num_levels; ++offset) {
+        def_level = def_levels[offset];
+
+        Roll(&def_level);
+        if (def_level == level_info_.def_level) {
+          RollValue(offset);
+        }
+        if (NeedNewChunk()) {
+          chunks.push_back({prev_offset, prev_offset, offset - prev_offset});
+          prev_offset = offset;
+        }
+      }
+      // set the previous value offset to add the last chunk
+      prev_value_offset = prev_offset;
+    } else {
+      // nested data with nulls
+      int16_t def_level;
+      int16_t rep_level;
+      int64_t value_offset = 0;
+
+      for (offset = 0; offset < num_levels; ++offset) {
+        def_level = def_levels[offset];
+        rep_level = rep_levels[offset];
+
+        Roll(&def_level);
+        Roll(&rep_level);
+        if (def_level == level_info_.def_level) {
+          RollValue(value_offset);
+        }
+
+        if ((rep_level == 0) && NeedNewChunk()) {
+          // if we are at a record boundary and need a new chunk, we create a new chunk
+          auto levels_to_write = offset - prev_offset;
+          if (levels_to_write > 0) {
+            chunks.push_back({prev_offset, prev_value_offset, levels_to_write});
+            prev_offset = offset;
+            prev_value_offset = value_offset;
+          }
+        }
+        if (def_level >= level_info_.repeated_ancestor_def_level) {
+          // we only increment the value offset if we have a leaf value
+          ++value_offset;
+        }
+      }
+    }
+
+    // add the last chunk if we have any levels left
+    if (prev_offset < num_levels) {
+      chunks.push_back({prev_offset, prev_value_offset, num_levels - prev_offset});
+    }
+#ifndef NDEBUG
+    ValidateChunks(chunks, num_levels);
+#endif
+
+    return chunks;
+  }
+
+  template <int kByteWidth>
+  std::vector<Chunk> CalculateFixedWidth(const int16_t* def_levels,
+                                         const int16_t* rep_levels, int64_t num_levels,
+                                         const ::arrow::Array& values) {
+    const uint8_t* raw_values =
+        values.data()->GetValues<uint8_t>(1, 0) + values.offset() * kByteWidth;
+    return Calculate(def_levels, rep_levels, num_levels, [&](int64_t i) {
+      return Roll<kByteWidth>(&raw_values[i * kByteWidth]);
+    });
+  }
+
+  template <typename ArrayType>
+  std::vector<Chunk> CalculateBinaryLike(const int16_t* def_levels,
+                                         const int16_t* rep_levels, int64_t num_levels,
+                                         const ::arrow::Array& values) {
+    const auto& array = checked_cast<const ArrayType&>(values);
+    const uint8_t* value;
+    typename ArrayType::offset_type length;
+    return Calculate(def_levels, rep_levels, num_levels, [&](int64_t i) {
+      value = array.GetValue(i, &length);
+      Roll(value, length);
+    });
+  }
+
+  std::vector<Chunk> GetChunks(const int16_t* def_levels, const int16_t* rep_levels,
+                               int64_t num_levels, const ::arrow::Array& values) {
+    auto handle_type = [&](auto&& type) -> std::vector<Chunk> {
+      using ArrowType = std::decay_t<decltype(type)>;
+      if constexpr (ArrowType::type_id == ::arrow::Type::NA) {
+        return Calculate(def_levels, rep_levels, num_levels, [](int64_t) {});
+      } else if constexpr (ArrowType::type_id == ::arrow::Type::BOOL) {
+        const auto& array = static_cast<const ::arrow::BooleanArray&>(values);
+        return Calculate(def_levels, rep_levels, num_levels,
+                         [&](int64_t i) { return Roll(array.Value(i)); });

Review Comment:
   Removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2026569435


##########
python/pyarrow/parquet/core.py:
##########
@@ -889,6 +889,39 @@ def _sanitize_table(table, new_schema, flavor):
     - fixed_len_byte_array: for precision > 18.
 
     As a consequence, decimal columns stored in integer types are more compact.
+use_content_defined_chunking : bool or dict, default False
+    Optimize parquet files for content addressable storage (CAS) systems by writing

Review Comment:
   Ah, sorry. No, we don't! Let's keep it like this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2026594849


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int8_t norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes
+  int64_t target_size = avg_chunk_size - min_chunk_size;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // 3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  if (norm_factor < -3 || norm_factor > 3) {
+    ARROW_LOG(WARNING) << "norm_factor=" << norm_factor
+                       << " is outside the recommended range [-3, 3]";
+  }
+
+  int mask_adjustement = 3 + norm_factor;
+  int effective_bits = mask_bits - mask_adjustement;
+  if (effective_bits == 0) {
+    return 0;

Review Comment:
   This was a theoretical limit for the validation, not a practical. I am making it more restrictive.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027849768


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();
+    auto min_length = ElementCount(min_chunk_size, byte_width, nullable);
+    auto max_length = ElementCount(max_chunk_size, byte_width, nullable);
+    AssertAllBetween(base_info.page_lengths, min_length, max_length);
+    AssertAllBetween(modified_info.page_lengths, min_length, max_length);
+  } else if (::arrow::is_base_binary_like(dtype->id()) && !enable_dictionary) {
+    AssertAllBetween(base_info.page_sizes, min_chunk_size, max_chunk_size);
+    AssertAllBetween(modified_info.page_sizes, min_chunk_size, max_chunk_size);
+  }
+}
+
+constexpr int64_t kMinChunkSize = 8 * 1024;
+constexpr int64_t kMaxChunkSize = 32 * 1024;
+constexpr int64_t kPartSize = 128 * 1024;
+constexpr int64_t kEditSize = 128;
+
+struct CaseConfig {
+  // Arrow data type to generate the testing data for
+  std::shared_ptr<::arrow::DataType> dtype;
+  // Whether the data type is nullable
+  bool is_nullable;
+  // Approximate number of bytes per record to calculate the number of elements to
+  // generate
+  size_t bytes_per_record;
+  // Data page version to use
+  ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1;
+};
+
+// Define PrintTo for MyStruct
+void PrintTo(const CaseConfig& param, std::ostream* os) {
+  *os << "{ " << param.dtype->ToString();
+  if (param.is_nullable) {
+    *os << " nullable";
+  }
+  *os << " }";
+}
+
+class TestCDC : public ::testing::Test {
+ public:
+  uint64_t GetMask(const ContentDefinedChunker& cdc) const { return cdc.GetMask(); }
+};
+
+TEST_F(TestCDC, RollingHashMaskCalculation) {
+  auto le = LevelInfo();
+  auto min_size = 256 * 1024;
+  auto max_size = 1024 * 1024;
+
+  auto cdc0 = ContentDefinedChunker(le, min_size, max_size, 0);
+  ASSERT_EQ(GetMask(cdc0), 0xFFFE000000000000);
+
+  auto cdc1 = ContentDefinedChunker(le, min_size, max_size, 1);
+  ASSERT_EQ(GetMask(cdc1), 0xFFFC000000000000);
+
+  auto cdc2 = ContentDefinedChunker(le, min_size, max_size, 2);
+  ASSERT_EQ(GetMask(cdc2), 0xFFF8000000000000);
+
+  auto cdc3 = ContentDefinedChunker(le, min_size, max_size, 3);
+  ASSERT_EQ(GetMask(cdc3), 0xFFF0000000000000);
+
+  auto cdc4 = ContentDefinedChunker(le, min_size, max_size, -1);
+  ASSERT_EQ(GetMask(cdc4), 0xFFFF000000000000);
+
+  // this is the smallest possible mask always matching, by using 8 hashtables
+  // we are going to have a match every 8 bytes; this is an unrealistic case
+  // but checking for the correctness of the mask calculation
+  auto cdc5 = ContentDefinedChunker(le, 0, 16, 0);
+  ASSERT_EQ(GetMask(cdc5), 0x0000000000000000);
+
+  auto cdc6 = ContentDefinedChunker(le, 0, 32, 1);
+  ASSERT_EQ(GetMask(cdc6), 0x0000000000000000);
+
+  auto cdc7 = ContentDefinedChunker(le, 0, 16, -1);
+  ASSERT_EQ(GetMask(cdc7), 0x8000000000000000);
+
+  // another unrealistic case, checking for the validation
+  auto cdc8 = ContentDefinedChunker(le, 128, 384, -60);
+  ASSERT_EQ(GetMask(cdc8), 0xFFFFFFFFFFFFFFFF);
+}
+
+TEST_F(TestCDC, WriteSingleColumnParquetFile) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  std::vector<int32_t> numbers = {1, 2, 3, 4, 5};
+  std::vector<uint8_t> valid_bits = {1, 0, 1, 0, 1};
+  EXPECT_THROW(
+      int_column_writer.WriteBatch(numbers.size(), nullptr, nullptr, numbers.data()),
+      ParquetException);

Review Comment:
   See https://github.com/apache/arrow/pull/45360#discussion_r1978502781



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027843805


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();
+    auto min_length = ElementCount(min_chunk_size, byte_width, nullable);
+    auto max_length = ElementCount(max_chunk_size, byte_width, nullable);
+    AssertAllBetween(base_info.page_lengths, min_length, max_length);
+    AssertAllBetween(modified_info.page_lengths, min_length, max_length);
+  } else if (::arrow::is_base_binary_like(dtype->id()) && !enable_dictionary) {
+    AssertAllBetween(base_info.page_sizes, min_chunk_size, max_chunk_size);
+    AssertAllBetween(modified_info.page_sizes, min_chunk_size, max_chunk_size);

Review Comment:
   Removed these checks, not relevant anymore.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027900181


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));
+    arrays.push_back(array);
+  }
+  return Table::Make(schema, arrays, size);
+}
+
+Result<std::shared_ptr<Table>> ConcatAndCombine(
+    const std::vector<std::shared_ptr<Table>>& parts) {
+  // Concatenate and combine chunks so the table doesn't carry information about
+  // the modification points
+  ARROW_ASSIGN_OR_RAISE(auto table, ConcatenateTables(parts));
+  return table->CombineChunks();
+}
+
+Result<std::shared_ptr<Buffer>> WriteTableToBuffer(
+    const std::shared_ptr<Table>& table, int64_t min_chunk_size, int64_t max_chunk_size,
+    bool enable_dictionary = false, int64_t row_group_size = 1024 * 1024,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1) {
+  auto sink = CreateOutputStream();
+
+  auto builder = WriterProperties::Builder();
+  builder.enable_content_defined_chunking()->content_defined_chunking_options(
+      min_chunk_size, max_chunk_size, /*norm_factor=*/0);
+  builder.data_page_version(data_page_version);
+  if (enable_dictionary) {
+    builder.enable_dictionary();
+  } else {
+    builder.disable_dictionary();
+  }
+  auto write_props = builder.build();
+  auto arrow_props = ArrowWriterProperties::Builder().store_schema()->build();
+  RETURN_NOT_OK(WriteTable(*table, default_memory_pool(), sink, row_group_size,
+                           write_props, arrow_props));
+  return sink->Finish();
+}
+
+Result<std::shared_ptr<Table>> ReadTableFromBuffer(const std::shared_ptr<Buffer>& data) {
+  std::shared_ptr<Table> result;
+  FileReaderBuilder builder;
+  std::unique_ptr<FileReader> reader;
+  RETURN_NOT_OK(builder.Open(std::make_shared<BufferReader>(data)));
+  RETURN_NOT_OK(builder.memory_pool(::arrow::default_memory_pool())
+                    ->properties(default_arrow_reader_properties())
+                    ->Build(&reader));
+  RETURN_NOT_OK(reader->ReadTable(&result));
+  return result;
+}
+
+// Type to represent a list of chunks where each element is the size of the chunk.
+using ChunkList = std::vector<int64_t>;
+
+// Type to represent the sizes and lengths of the data pages in a column.
+
+struct RowGroupInfo {
+  ChunkList page_lengths;
+  ChunkList page_sizes;
+  bool has_dictionary_page = false;
+};
+
+using ParquetInfo = std::vector<RowGroupInfo>;
+
+ParquetInfo GetColumnParquetInfo(const std::shared_ptr<Buffer>& data,
+                                 int column_index = 0) {
+  // Read the parquet data out of the buffer and get the sizes and lengths of the
+  // data pages in given column. We assert on the sizes and lengths of the pages
+  // to ensure that the chunking is done correctly.
+  ParquetInfo result;
+
+  auto buffer_reader = std::make_shared<BufferReader>(data);
+  auto parquet_reader = ParquetFileReader::Open(std::move(buffer_reader));
+
+  auto metadata = parquet_reader->metadata();
+  for (int rg = 0; rg < metadata->num_row_groups(); rg++) {
+    auto page_reader = parquet_reader->RowGroup(rg)->GetColumnPageReader(column_index);
+    RowGroupInfo rg_info;
+    while (auto page = page_reader->NextPage()) {
+      if (page->type() == PageType::DATA_PAGE || page->type() == PageType::DATA_PAGE_V2) {
+        auto data_page = static_cast<DataPage*>(page.get());
+        rg_info.page_sizes.push_back(data_page->size());
+        rg_info.page_lengths.push_back(data_page->num_values());
+      } else if (page->type() == PageType::DICTIONARY_PAGE) {
+        rg_info.has_dictionary_page = true;
+      }
+    }
+    result.push_back(rg_info);
+  }
+
+  return result;
+}
+
+Result<ParquetInfo> WriteAndGetParquetInfo(
+    const std::shared_ptr<Table>& table, uint64_t min_chunk_size, uint64_t max_chunk_size,
+    bool enable_dictionary = false,
+    ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1,
+    int64_t row_group_size = 1024 * 1024,
+
+    int column_index = 0) {
+  // Write the table to a buffer and read it back to get the page sizes
+  ARROW_ASSIGN_OR_RAISE(
+      auto buffer,
+      WriteTableToBuffer(table, min_chunk_size, max_chunk_size, enable_dictionary,
+                         row_group_size, data_page_version));
+  ARROW_ASSIGN_OR_RAISE(auto readback, ReadTableFromBuffer(buffer));
+
+  RETURN_NOT_OK(readback->ValidateFull());
+  if (readback->schema()->Equals(*table->schema())) {
+    ARROW_RETURN_IF(!readback->Equals(*table),
+                    Status::Invalid("Readback table not equal to original"));
+  }
+  return GetColumnParquetInfo(buffer, column_index);
+}
+
+// A git-hunk like side-by-side data structure to represent the differences between two
+// vectors of uint64_t values.
+using ChunkDiff = std::pair<ChunkList, ChunkList>;
+
+/**
+ * Finds the differences between two sequences of chunk lengths or sizes.
+ * Uses a longest common subsequence algorithm to identify matching elements
+ * and extract the differences between the sequences.
+ *
+ * @param first The first sequence of chunk values
+ * @param second The second sequence of chunk values
+ * @return A vector of differences, where each difference is a pair of
+ *         subsequences (one from each input) that differ
+ */
+std::vector<ChunkDiff> FindDifferences(const ChunkList& first, const ChunkList& second) {
+  // Compute the longest common subsequence using dynamic programming
+  size_t n = first.size(), m = second.size();
+  std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1, 0));
+
+  // Fill the dynamic programming table
+  for (size_t i = 0; i < n; i++) {
+    for (size_t j = 0; j < m; j++) {
+      if (first[i] == second[j]) {
+        // If current elements match, extend the LCS
+        dp[i + 1][j + 1] = dp[i][j] + 1;
+      } else {
+        // If current elements don't match, take the best option
+        dp[i + 1][j + 1] = std::max(dp[i + 1][j], dp[i][j + 1]);
+      }
+    }
+  }
+
+  // Backtrack through the dynamic programming table to reconstruct the common
+  // parts and their positions in the original sequences
+  std::vector<std::pair<size_t, size_t>> common;
+  for (size_t i = n, j = m; i > 0 && j > 0;) {
+    if (first[i - 1] == second[j - 1]) {
+      // Found a common element, add to common list
+      common.emplace_back(i - 1, j - 1);
+      i--, j--;
+    } else if (dp[i - 1][j] >= dp[i][j - 1]) {
+      // Move in the direction of the larger LCS value
+      i--;
+    } else {
+      j--;
+    }
+  }
+  // Reverse to get indices in ascending order
+  std::reverse(common.begin(), common.end());
+
+  // Build the differences by finding sequences between common elements
+  std::vector<ChunkDiff> result;
+  size_t last_i = 0, last_j = 0;
+  for (auto& c : common) {
+    auto ci = c.first;
+    auto cj = c.second;
+    // If there's a gap between the last common element and this one,
+    // record the difference
+    if (ci > last_i || cj > last_j) {
+      result.push_back({{first.begin() + last_i, first.begin() + ci},
+                        {second.begin() + last_j, second.begin() + cj}});
+    }
+    // Move past this common element
+    last_i = ci + 1;
+    last_j = cj + 1;
+  }
+
+  // Handle any remaining elements after the last common element
+  if (last_i < n || last_j < m) {
+    result.push_back(
+        {{first.begin() + last_i, first.end()}, {second.begin() + last_j, second.end()}});
+  }
+
+  // Post-process: merge adjacent diffs to avoid splitting single changes into multiple
+  // parts
+  std::vector<ChunkDiff> merged;
+  for (auto& diff : result) {
+    if (!merged.empty()) {
+      auto& prev = merged.back();
+      // Check if we can merge with the previous diff
+      bool can_merge_a = prev.first.empty() && !prev.second.empty() &&
+                         !diff.first.empty() && diff.second.empty();
+      bool can_merge_b = prev.second.empty() && !prev.first.empty() &&
+                         !diff.second.empty() && diff.first.empty();
+
+      if (can_merge_a) {
+        // Combine into one diff: keep prev's second, use diff's first
+        prev.first = std::move(diff.first);
+        continue;
+      } else if (can_merge_b) {
+        // Combine into one diff: keep prev's first, use diff's second
+        prev.second = std::move(diff.second);
+        continue;
+      }
+    }
+    // If we can't merge, add this diff to the result
+    merged.push_back(std::move(diff));
+  }
+  return merged;
+}
+
+void PrintDifferences(const ChunkList& original, const ChunkList& modified,
+                      std::vector<ChunkDiff>& diffs) {
+  // Utility function to print the original and modified sequences, and the diffs
+  // between them. Used in case of failing assertions to display the differences.
+  std::cout << "Original: ";
+  for (const auto& val : original) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  std::cout << "Modified: ";
+  for (const auto& val : modified) {
+    std::cout << val << " ";
+  }
+  std::cout << std::endl;
+
+  for (const auto& diff : diffs) {
+    std::cout << "First: ";
+    for (const auto& val : diff.first) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+
+    std::cout << "Second: ";
+    for (const auto& val : diff.second) {
+      std::cout << val << " ";
+    }
+    std::cout << std::endl;
+  }
+}
+
+TEST(TestFindDifferences, Basic) {
+  ChunkList first = {1, 2, 3, 4, 5};
+  ChunkList second = {1, 7, 8, 4, 5};
+
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({7, 8}));
+}
+
+TEST(TestFindDifferences, MultipleDifferences) {
+  ChunkList first = {1, 2, 3, 4, 5, 6, 7};
+  ChunkList second = {1, 8, 9, 4, 10, 6, 11};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 3);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({2, 3}));
+  ASSERT_EQ(diffs[0].second, ChunkList({8, 9}));
+
+  ASSERT_EQ(diffs[1].first, ChunkList({5}));
+  ASSERT_EQ(diffs[1].second, ChunkList({10}));
+
+  ASSERT_EQ(diffs[2].first, ChunkList({7}));
+  ASSERT_EQ(diffs[2].second, ChunkList({11}));
+}
+
+TEST(TestFindDifferences, DifferentLengths) {
+  ChunkList first = {1, 2, 3};
+  ChunkList second = {1, 2, 3, 4, 5};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_TRUE(diffs[0].first.empty());
+  ASSERT_EQ(diffs[0].second, ChunkList({4, 5}));
+}
+
+TEST(TestFindDifferences, EmptyArrays) {
+  ChunkList first = {};
+  ChunkList second = {};
+  auto diffs = FindDifferences(first, second);
+  ASSERT_TRUE(diffs.empty());
+}
+
+TEST(TestFindDifferences, LongSequenceWithSingleDifference) {
+  ChunkList first = {
+      1994, 2193, 2700, 1913, 2052,
+  };
+  ChunkList second = {2048, 43, 2080, 2700, 1913, 2052};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1994, 2193}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2048, 43, 2080}));
+
+  // Verify that elements after the difference are identical
+  for (size_t i = 3; i < second.size(); i++) {
+    ASSERT_EQ(first[i - 1], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, LongSequenceWithMiddleChanges) {
+  ChunkList first = {2169, 1976, 2180, 2147, 1934, 1772,
+                     1914, 2075, 2154, 1940, 1934, 1970};
+  ChunkList second = {2169, 1976, 2180, 2147, 2265, 1804,
+                      1717, 1925, 2122, 1940, 1934, 1970};
+  auto diffs = FindDifferences(first, second);
+
+  ASSERT_EQ(diffs.size(), 1);
+  ASSERT_EQ(diffs[0].first, ChunkList({1934, 1772, 1914, 2075, 2154}));
+  ASSERT_EQ(diffs[0].second, ChunkList({2265, 1804, 1717, 1925, 2122}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 4; i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+  for (size_t i = 9; i < first.size(); i++) {
+    ASSERT_EQ(first[i], second[i]);
+  }
+}
+
+TEST(TestFindDifferences, AdditionalCase) {
+  ChunkList original = {445, 312, 393, 401, 410, 138, 558, 457};
+  ChunkList modified = {445, 312, 393, 393, 410, 138, 558, 457};
+
+  auto diffs = FindDifferences(original, modified);
+  ASSERT_EQ(diffs.size(), 1);
+
+  ASSERT_EQ(diffs[0].first, ChunkList({401}));
+  ASSERT_EQ(diffs[0].second, ChunkList({393}));
+
+  // Verify elements before and after the difference are identical
+  for (size_t i = 0; i < 3; i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+  for (size_t i = 4; i < original.size(); i++) {
+    ASSERT_EQ(original[i], modified[i]);
+  }
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 int8_t exact_number_of_equal_diffs,
+                                 int8_t exact_number_of_larger_diffs,
+                                 int8_t exact_number_of_smaller_diffs,
+                                 int64_t edit_length = 0) {
+  // Asserts that the differences between the original and modified page lengths
+  // are as expected. A longest common subsequence diff is calculated on the original
+  // and modified sequences of page lengths. The exact_number_of_equal_diffs,
+  // exact_number_of_larger_diffs, and exact_number_of_smaller_diffs parameters specify
+  // the expected number of differences with equal, larger, and smaller sums of the page
+  // lengths respectively. The edit_length parameter is used to verify that the page
+  // lenght differences are exactly equal to the edit_length.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  size_t expected_number_of_diffs = exact_number_of_equal_diffs +
+                                    exact_number_of_larger_diffs +
+                                    exact_number_of_smaller_diffs;
+  if (diffs.size() != expected_number_of_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+  ASSERT_EQ(diffs.size(), expected_number_of_diffs);
+
+  uint8_t equal_diffs = 0;
+  int8_t larger_diffs = 0;
+  int8_t smaller_diffs = 0;
+  for (const auto& diff : diffs) {
+    uint64_t original_sum = 0, modified_sum = 0;
+    for (const auto& val : diff.first) original_sum += val;
+    for (const auto& val : diff.second) modified_sum += val;
+
+    if (original_sum == modified_sum) {
+      equal_diffs++;
+    } else if (original_sum < modified_sum) {
+      larger_diffs++;
+      ASSERT_EQ(original_sum + edit_length, modified_sum);
+    } else if (original_sum > modified_sum) {
+      smaller_diffs++;
+      ASSERT_EQ(original_sum, modified_sum + edit_length);
+    }
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  ASSERT_EQ(equal_diffs, exact_number_of_equal_diffs);
+  ASSERT_EQ(larger_diffs, exact_number_of_larger_diffs);
+  ASSERT_EQ(smaller_diffs, exact_number_of_smaller_diffs);
+}
+
+void AssertPageLengthDifferences(const RowGroupInfo& original,
+                                 const RowGroupInfo& modified,
+                                 uint8_t max_number_of_equal_diffs) {
+  // A less restrictive version of the above assertion function mainly used to
+  // assert the update case.
+  auto diffs = FindDifferences(original.page_lengths, modified.page_lengths);
+  if (diffs.size() > max_number_of_equal_diffs) {
+    PrintDifferences(original.page_lengths, modified.page_lengths, diffs);
+  }
+  ASSERT_LE(diffs.size(), max_number_of_equal_diffs);
+
+  for (const auto& diff : diffs) {
+    uint64_t left_sum = 0, right_sum = 0;
+    for (const auto& val : diff.first) left_sum += val;
+    for (const auto& val : diff.second) right_sum += val;
+    ASSERT_EQ(left_sum, right_sum);
+    ASSERT_LE(diff.first.size(), 2);
+    ASSERT_LE(diff.second.size(), 2);
+  }
+
+  if (diffs.size() == 0) {
+    // no differences found, the arrays are equal
+    ASSERT_TRUE(original.page_lengths == modified.page_lengths);
+  }
+}
+
+uint64_t ElementCount(int64_t size, int32_t byte_width, bool nullable) {
+  if (nullable) {
+    // in case of nullable types the def_levels are also fed through the chunker
+    // to identify changes in the null bitmap, this will increase the byte width
+    // and decrease the number of elements per chunk
+    byte_width += 2;
+  }
+  return size / byte_width;
+}
+
+void AssertAllBetween(const ChunkList& chunks, int64_t min, int64_t max) {
+  // except the last chunk since it is not guaranteed to be within the range
+  for (size_t i = 0; i < chunks.size() - 1; i++) {
+    ASSERT_GE(chunks[i], min);
+    ASSERT_LE(chunks[i], max);
+  }
+  ASSERT_LE(chunks.back(), max);
+}
+
+void AssertChunkSizes(const std::shared_ptr<::arrow::DataType>& dtype,
+                      const RowGroupInfo& base_info, const RowGroupInfo& modified_info,
+                      bool nullable, bool enable_dictionary, int64_t min_chunk_size,
+                      int64_t max_chunk_size) {
+  if (dtype->id() != ::arrow::Type::BOOL) {
+    ASSERT_EQ(base_info.has_dictionary_page, enable_dictionary);
+    ASSERT_EQ(modified_info.has_dictionary_page, enable_dictionary);
+  }
+  if (::arrow::is_fixed_width(dtype->id())) {
+    // for nullable types we cannot calculate the exact number of elements because
+    // not all elements are fed through the chunker (null elements are skipped)
+    auto byte_width = (dtype->id() == ::arrow::Type::BOOL) ? 1 : dtype->byte_width();
+    auto min_length = ElementCount(min_chunk_size, byte_width, nullable);
+    auto max_length = ElementCount(max_chunk_size, byte_width, nullable);
+    AssertAllBetween(base_info.page_lengths, min_length, max_length);
+    AssertAllBetween(modified_info.page_lengths, min_length, max_length);
+  } else if (::arrow::is_base_binary_like(dtype->id()) && !enable_dictionary) {
+    AssertAllBetween(base_info.page_sizes, min_chunk_size, max_chunk_size);
+    AssertAllBetween(modified_info.page_sizes, min_chunk_size, max_chunk_size);
+  }
+}
+
+constexpr int64_t kMinChunkSize = 8 * 1024;
+constexpr int64_t kMaxChunkSize = 32 * 1024;
+constexpr int64_t kPartSize = 128 * 1024;
+constexpr int64_t kEditSize = 128;
+
+struct CaseConfig {
+  // Arrow data type to generate the testing data for
+  std::shared_ptr<::arrow::DataType> dtype;
+  // Whether the data type is nullable
+  bool is_nullable;
+  // Approximate number of bytes per record to calculate the number of elements to
+  // generate
+  size_t bytes_per_record;
+  // Data page version to use
+  ParquetDataPageVersion data_page_version = ParquetDataPageVersion::V1;
+};
+
+// Define PrintTo for MyStruct
+void PrintTo(const CaseConfig& param, std::ostream* os) {
+  *os << "{ " << param.dtype->ToString();
+  if (param.is_nullable) {
+    *os << " nullable";
+  }
+  *os << " }";
+}
+
+class TestCDC : public ::testing::Test {
+ public:
+  uint64_t GetMask(const ContentDefinedChunker& cdc) const { return cdc.GetMask(); }
+};
+
+TEST_F(TestCDC, RollingHashMaskCalculation) {
+  auto le = LevelInfo();
+  auto min_size = 256 * 1024;
+  auto max_size = 1024 * 1024;
+
+  auto cdc0 = ContentDefinedChunker(le, min_size, max_size, 0);
+  ASSERT_EQ(GetMask(cdc0), 0xFFFE000000000000);
+
+  auto cdc1 = ContentDefinedChunker(le, min_size, max_size, 1);
+  ASSERT_EQ(GetMask(cdc1), 0xFFFC000000000000);
+
+  auto cdc2 = ContentDefinedChunker(le, min_size, max_size, 2);
+  ASSERT_EQ(GetMask(cdc2), 0xFFF8000000000000);
+
+  auto cdc3 = ContentDefinedChunker(le, min_size, max_size, 3);
+  ASSERT_EQ(GetMask(cdc3), 0xFFF0000000000000);
+
+  auto cdc4 = ContentDefinedChunker(le, min_size, max_size, -1);
+  ASSERT_EQ(GetMask(cdc4), 0xFFFF000000000000);
+
+  // this is the smallest possible mask always matching, by using 8 hashtables
+  // we are going to have a match every 8 bytes; this is an unrealistic case
+  // but checking for the correctness of the mask calculation
+  auto cdc5 = ContentDefinedChunker(le, 0, 16, 0);
+  ASSERT_EQ(GetMask(cdc5), 0x0000000000000000);
+
+  auto cdc6 = ContentDefinedChunker(le, 0, 32, 1);
+  ASSERT_EQ(GetMask(cdc6), 0x0000000000000000);
+
+  auto cdc7 = ContentDefinedChunker(le, 0, 16, -1);
+  ASSERT_EQ(GetMask(cdc7), 0x8000000000000000);
+
+  // another unrealistic case, checking for the validation
+  auto cdc8 = ContentDefinedChunker(le, 128, 384, -60);
+  ASSERT_EQ(GetMask(cdc8), 0xFFFFFFFFFFFFFFFF);
+}
+
+TEST_F(TestCDC, WriteSingleColumnParquetFile) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  std::vector<int32_t> numbers = {1, 2, 3, 4, 5};
+  std::vector<uint8_t> valid_bits = {1, 0, 1, 0, 1};
+  EXPECT_THROW(
+      int_column_writer.WriteBatch(numbers.size(), nullptr, nullptr, numbers.data()),
+      ParquetException);
+  EXPECT_THROW(int_column_writer.WriteBatchSpaced(numbers.size(), nullptr, nullptr,
+                                                  valid_bits.data(), 0, numbers.data()),
+               ParquetException);
+}
+
+TEST_F(TestCDC, LastChunkDoesntTriggerAddDataPage) {
+  // Define the schema with a single column "number"
+  auto schema = std::dynamic_pointer_cast<schema::GroupNode>(schema::GroupNode::Make(
+      "root", Repetition::REQUIRED,
+      {schema::PrimitiveNode::Make("number", Repetition::REQUIRED, Type::INT32)}));
+
+  auto sink = CreateOutputStream();
+  auto builder = WriterProperties::Builder();
+  auto props = builder.enable_content_defined_chunking()
+                   ->content_defined_chunking_options(kMinChunkSize, kMaxChunkSize, 0)
+                   ->disable_dictionary()
+                   ->build();
+
+  auto writer = ParquetFileWriter::Open(sink, schema, props);
+  auto row_group_writer = writer->AppendRowGroup();
+
+  // Create a column writer for the "number" column
+  auto column_writer = row_group_writer->NextColumn();
+  auto& int_column_writer = dynamic_cast<Int32Writer&>(*column_writer);
+
+  ASSERT_OK_AND_ASSIGN(auto array, ::arrow::gen::Step()->Generate(8000));
+  auto arrow_props = default_arrow_writer_properties();
+  auto arrow_ctx = ArrowWriteContext(default_memory_pool(), arrow_props.get());
+
+  // Calling WriteArrow twice, we expect that the first call doesn't add a new data page
+  // at the end allowing subsequent calls to append to the same page
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+  ASSERT_OK(int_column_writer.WriteArrow(nullptr, nullptr, array->length(), *array,
+                                         &arrow_ctx, false));
+
+  int_column_writer.Close();
+  writer->Close();
+  ASSERT_OK_AND_ASSIGN(auto buffer, sink->Finish());
+
+  auto info = GetColumnParquetInfo(buffer);
+  ASSERT_EQ(info.size(), 1);
+
+  // AssertAllBetween allow the last chunk size to be smaller than the min_chunk_size
+  AssertAllBetween(info[0].page_sizes, kMinChunkSize, kMaxChunkSize);
+  AssertAllBetween(info[0].page_lengths, 3000, 5000);
+}
+
+TEST_F(TestCDC, ChunkSizeParameterValidation) {
+  // Test that constructor validates min/max chunk size parameters
+  auto li = LevelInfo();
+
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 256 * 1024, 1024 * 1024));
+
+  // with norm_factor=0 the difference between min and max chunk size must be
+  // at least 16
+  ASSERT_THROW(ContentDefinedChunker(li, 0, -1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1024, 512), ParquetException);
+
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16));
+  ASSERT_THROW(ContentDefinedChunker(li, -16, -16), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 16, 0), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 32, 32), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 32, 48));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1024 * 1024, 2 * 1024 * 1024));
+  ASSERT_NO_THROW(
+      ContentDefinedChunker(li, 1024 * 1024 * 1024L, 2LL * 1024 * 1024 * 1024L));
+
+  // with norm_factor=1 the difference between min and max chunk size must be
+  // at least 64
+  ASSERT_THROW(ContentDefinedChunker(li, 1, -1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, -1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 1, 1), ParquetException);
+  ASSERT_THROW(ContentDefinedChunker(li, 1, 32, 1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 1, 33, 1));
+
+  // with norm_factor=2 the difference between min and max chunk size must be
+  // at least 128
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 63, 2), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 64, 2));
+
+  // with norm_factor=-1 the difference between min and max chunk size must be
+  // at least 8
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 7, -1), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 8, -1));
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 16, -1));
+
+  // test the norm_factor extremes
+  ASSERT_THROW(ContentDefinedChunker(li, 0, 0, -68), ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, 1, -67));
+  ASSERT_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 59),
+               ParquetException);
+  ASSERT_NO_THROW(ContentDefinedChunker(li, 0, std::numeric_limits<int64_t>::max(), 58));
+}
+
+class TestCDCSingleRowGroup : public ::testing::TestWithParam<CaseConfig> {
+ protected:
+  // Column random table parts for testing
+  std::shared_ptr<Field> field_;
+  std::shared_ptr<Table> part1_, part2_, part3_, part4_, part5_, part6_, part7_;
+
+  void SetUp() override {
+    const auto& param = GetParam();
+    auto field_ = ::arrow::field("f0", param.dtype, param.is_nullable);
+    auto schema = ::arrow::schema({field_});
+
+    auto part_length = kPartSize / param.bytes_per_record;
+    auto edit_length = kEditSize / param.bytes_per_record;
+    ASSERT_OK_AND_ASSIGN(part1_, GenerateTable(schema, part_length, 0));
+    ASSERT_OK_AND_ASSIGN(part2_, GenerateTable(schema, edit_length, 1));
+    ASSERT_OK_AND_ASSIGN(part3_, GenerateTable(schema, part_length, part_length));
+    ASSERT_OK_AND_ASSIGN(part4_, GenerateTable(schema, edit_length, 2));
+    ASSERT_OK_AND_ASSIGN(part5_, GenerateTable(schema, part_length, 2 * part_length));
+    ASSERT_OK_AND_ASSIGN(part6_, GenerateTable(schema, edit_length, 3));
+    ASSERT_OK_AND_ASSIGN(part7_, GenerateTable(schema, edit_length, 4));
+  }
+};
+
+TEST_P(TestCDCSingleRowGroup, DeleteOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/0,
+                                /*exact_number_of_smaller_diffs=*/1, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, DeleteTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part3_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/0,
+                                /*exact_number_of_smaller_diffs=*/2, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, UpdateOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part4_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*max_number_of_equal_diffs=*/1);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, UpdateTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified,
+                       ConcatAndCombine({part1_, part6_, part3_, part7_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*max_number_of_equal_diffs=*/2);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, InsertOnce) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/1,
+                                /*exact_number_of_smaller_diffs=*/0, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, InsertTwice) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part3_, part5_}));
+  ASSERT_OK_AND_ASSIGN(auto modified,
+                       ConcatAndCombine({part1_, part2_, part3_, part4_, part5_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto edit_length = part2_->num_rows();
+    if (::arrow::is_list_like(param.dtype->id())) {
+      edit_length += 1;
+    }
+    AssertPageLengthDifferences(base_info.front(), modified_info.front(),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/2,
+                                /*exact_number_of_smaller_diffs=*/0, edit_length);
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, Append) {
+  const auto& param = GetParam();
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto modified, ConcatAndCombine({part1_, part2_, part3_, part4_}));
+  ASSERT_FALSE(base->Equals(*modified));
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(auto base_info, WriteAndGetParquetInfo(
+                                             base, kMinChunkSize, kMaxChunkSize,
+                                             enable_dictionary, param.data_page_version));
+    ASSERT_OK_AND_ASSIGN(
+        auto modified_info,
+        WriteAndGetParquetInfo(modified, kMinChunkSize, kMaxChunkSize, enable_dictionary,
+                               param.data_page_version));
+    // assert that there is only one row group
+    ASSERT_EQ(base_info.size(), 1);
+    ASSERT_EQ(modified_info.size(), 1);
+    AssertChunkSizes(param.dtype, base_info.front(), modified_info.front(),
+                     param.is_nullable, enable_dictionary, kMinChunkSize, kMaxChunkSize);
+
+    auto original_page_lengths = base_info.front().page_lengths;
+    auto modified_page_lengths = modified_info.front().page_lengths;
+    ASSERT_GE(original_page_lengths.size(), modified_page_lengths.size());
+    for (size_t i = 0; i < original_page_lengths.size() - 1; i++) {
+      ASSERT_EQ(original_page_lengths[i], modified_page_lengths[i]);
+    }
+    ASSERT_GT(modified_page_lengths.back(), original_page_lengths.back());
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, EmptyTable) {
+  const auto& param = GetParam();
+
+  auto schema = ::arrow::schema({::arrow::field("f0", param.dtype, param.is_nullable)});
+  ASSERT_OK_AND_ASSIGN(auto empty_table, GenerateTable(schema, 0, 0));
+  ASSERT_EQ(empty_table->num_rows(), 0);
+
+  for (bool enable_dictionary : {false, true}) {
+    ASSERT_OK_AND_ASSIGN(
+        auto result, WriteAndGetParquetInfo(empty_table, kMinChunkSize, kMaxChunkSize,
+                                            enable_dictionary, param.data_page_version));
+
+    // An empty table should result in no data pages
+    ASSERT_EQ(result.size(), 1);
+    ASSERT_TRUE(result.front().page_lengths.empty());
+    ASSERT_TRUE(result.front().page_sizes.empty());
+  }
+}
+
+TEST_P(TestCDCSingleRowGroup, ArrayOffsets) {
+  const auto& param = GetParam();
+  ASSERT_OK_AND_ASSIGN(auto table, ConcatAndCombine({part1_, part2_, part3_}));
+
+  for (auto offset : {0, 512, 1024}) {
+    auto sliced_table = table->Slice(offset);
+
+    // assert that the first column has a non-zero offset
+    auto column = sliced_table->column(0);
+    auto first_chunk = column->chunk(0);
+    ASSERT_EQ(first_chunk->offset(), offset);
+
+    // write out the sliced table, read it back and compare
+    ASSERT_OK(WriteAndGetParquetInfo(sliced_table, kMinChunkSize, kMaxChunkSize, true,
+                                     param.data_page_version));
+  }
+}
+
+INSTANTIATE_TEST_SUITE_P(
+    FixedSizedTypes, TestCDCSingleRowGroup,
+    testing::Values(
+        // Boolean
+        CaseConfig{::arrow::boolean(), false, 1},
+        // Numeric
+        CaseConfig{::arrow::uint8(), false, 1}, CaseConfig{::arrow::uint16(), false, 2},
+        CaseConfig{::arrow::uint32(), false, 4}, CaseConfig{::arrow::uint64(), true, 8},
+        CaseConfig{::arrow::int8(), false, 1}, CaseConfig{::arrow::int16(), false, 2},
+        CaseConfig{::arrow::int32(), false, 4}, CaseConfig{::arrow::int64(), true, 8},
+        CaseConfig{::arrow::float16(), false, 2},
+        CaseConfig{::arrow::float32(), false, 4}, CaseConfig{::arrow::float64(), true, 8},
+        CaseConfig{::arrow::decimal128(18, 6), false, 16},
+        CaseConfig{::arrow::decimal256(40, 6), false, 32},
+        // Binary-like
+        CaseConfig{::arrow::utf8(), false, 16}, CaseConfig{::arrow::binary(), true, 16},
+        CaseConfig{::arrow::fixed_size_binary(16), true, 16},
+        // Temporal
+        CaseConfig{::arrow::date32(), false, 4},
+        CaseConfig{::arrow::time32(::arrow::TimeUnit::MILLI), true, 4},
+        CaseConfig{::arrow::time64(::arrow::TimeUnit::NANO), false, 8},
+        CaseConfig{::arrow::timestamp(::arrow::TimeUnit::NANO), true, 8},
+        CaseConfig{::arrow::duration(::arrow::TimeUnit::NANO), false, 8},
+        // Nested types
+        CaseConfig{::arrow::list(::arrow::int32()), false, 16},
+        CaseConfig{::arrow::list(::arrow::int32()), true, 18},
+        CaseConfig{::arrow::list(::arrow::utf8()), true, 18},
+        CaseConfig{::arrow::struct_({::arrow::field("f0", ::arrow::int32())}), false, 8},
+        CaseConfig{::arrow::struct_({::arrow::field("f0", ::arrow::float64())}), true,
+                   10},
+        CaseConfig{
+            ::arrow::list(::arrow::struct_({::arrow::field("f0", ::arrow::int32())})),
+            false, 16},
+        // Extension type
+        CaseConfig{::arrow::uuid(), true, 16},
+        // Use ParquetDataPageVersion::V2
+        CaseConfig{::arrow::large_binary(), false, 16, ParquetDataPageVersion::V2},
+        CaseConfig{::arrow::list(::arrow::utf8()), true, 18,
+                   ParquetDataPageVersion::V2}));
+
+class TestCDCMultipleRowGroups : public ::testing::Test {
+ protected:
+  // Column random table parts for testing
+  std::shared_ptr<DataType> dtype_;
+  std::shared_ptr<Table> part1_, part2_, part3_;
+  std::shared_ptr<Table> edit1_, edit2_, edit3_;
+
+  void SetUp() override {
+    auto constexpr kPartLength = 256 * 1024;
+    auto constexpr kEditLength = 128;
+
+    dtype_ = ::arrow::int32();
+    auto field = ::arrow::field("f0", dtype_, true);
+    auto schema = ::arrow::schema({field});
+
+    ASSERT_OK_AND_ASSIGN(part1_, GenerateTable(schema, kPartLength, 0));
+    ASSERT_OK_AND_ASSIGN(part2_, GenerateTable(schema, kPartLength, 2));
+    ASSERT_OK_AND_ASSIGN(part3_, GenerateTable(schema, kPartLength, 4));
+
+    ASSERT_OK_AND_ASSIGN(edit1_, GenerateTable(schema, kEditLength, 1));
+    ASSERT_OK_AND_ASSIGN(edit2_, GenerateTable(schema, kEditLength, 3));
+    ASSERT_OK_AND_ASSIGN(edit3_, GenerateTable(schema, kEditLength, 5));
+  }
+};
+
+TEST_F(TestCDCMultipleRowGroups, InsertOnce) {
+  auto constexpr kRowGroupLength = 128 * 1024;
+  auto constexpr kEnableDictionary = false;
+  auto constexpr kMinChunkSize = 0 * 1024;
+  auto constexpr kMaxChunkSize = 128 * 1024;
+
+  ASSERT_OK_AND_ASSIGN(auto base, ConcatAndCombine({part1_, edit1_, part2_, part3_}));
+  ASSERT_OK_AND_ASSIGN(auto inserted,
+                       ConcatAndCombine({part1_, edit1_, edit2_, part2_, part3_}));
+  ASSERT_FALSE(base->Equals(*inserted));
+  ASSERT_EQ(inserted->num_rows(), base->num_rows() + edit2_->num_rows());
+
+  ASSERT_OK_AND_ASSIGN(
+      auto base_info,
+      WriteAndGetParquetInfo(base, kMinChunkSize, kMaxChunkSize, kEnableDictionary,
+                             ParquetDataPageVersion::V1, kRowGroupLength));
+  ASSERT_OK_AND_ASSIGN(
+      auto inserted_info,
+      WriteAndGetParquetInfo(inserted, kMinChunkSize, kMaxChunkSize, kEnableDictionary,
+                             ParquetDataPageVersion::V1, kRowGroupLength));
+
+  ASSERT_EQ(base_info.size(), 7);
+  ASSERT_EQ(inserted_info.size(), 7);
+
+  ASSERT_EQ(base_info.at(0).page_lengths, inserted_info.at(0).page_lengths);
+  ASSERT_EQ(base_info.at(1).page_lengths, inserted_info.at(1).page_lengths);
+  for (size_t i = 2; i < inserted_info.size() - 1; i++) {
+    AssertPageLengthDifferences(base_info.at(i), inserted_info.at(i),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/1,
+                                /*exact_number_of_smaller_diffs=*/1, edit2_->num_rows());
+  }
+  AssertPageLengthDifferences(base_info.back(), inserted_info.back(),
+                              /*exact_number_of_equal_diffs=*/0,
+                              /*exact_number_of_larger_diffs=*/1,
+                              /*exact_number_of_smaller_diffs=*/0, edit2_->num_rows());
+}
+
+TEST_F(TestCDCMultipleRowGroups, DeleteOnce) {
+  auto constexpr kRowGroupLength = 128 * 1024;
+  auto constexpr kEnableDictionary = false;
+  auto constexpr kMinChunkSize = 0 * 1024;
+  auto constexpr kMaxChunkSize = 128 * 1024;
+
+  ASSERT_OK_AND_ASSIGN(auto base,
+                       ConcatAndCombine({part1_, edit1_, part2_, part3_, edit2_}));
+  ASSERT_OK_AND_ASSIGN(auto deleted, ConcatAndCombine({part1_, part2_, part3_, edit2_}));
+  ASSERT_FALSE(base->Equals(*deleted));
+  ASSERT_EQ(deleted->num_rows(), base->num_rows() - edit1_->num_rows());
+
+  ASSERT_OK_AND_ASSIGN(
+      auto base_info,
+      WriteAndGetParquetInfo(base, kMinChunkSize, kMaxChunkSize, kEnableDictionary,
+                             ParquetDataPageVersion::V1, kRowGroupLength));
+  ASSERT_OK_AND_ASSIGN(
+      auto deleted_info,
+      WriteAndGetParquetInfo(deleted, kMinChunkSize, kMaxChunkSize, kEnableDictionary,
+                             ParquetDataPageVersion::V1, kRowGroupLength));
+
+  ASSERT_EQ(base_info.size(), 7);
+  ASSERT_EQ(deleted_info.size(), 7);
+
+  ASSERT_EQ(base_info.at(0).page_lengths, deleted_info.at(0).page_lengths);
+  ASSERT_EQ(base_info.at(1).page_lengths, deleted_info.at(1).page_lengths);
+  for (size_t i = 2; i < deleted_info.size() - 1; i++) {
+    AssertPageLengthDifferences(base_info.at(i), deleted_info.at(i),
+                                /*exact_number_of_equal_diffs=*/0,
+                                /*exact_number_of_larger_diffs=*/1,
+                                /*exact_number_of_smaller_diffs=*/1, edit1_->num_rows());
+  }
+  AssertPageLengthDifferences(base_info.back(), deleted_info.back(),
+                              /*exact_number_of_equal_diffs=*/0,
+                              /*exact_number_of_larger_diffs=*/0,
+                              /*exact_number_of_smaller_diffs=*/1, edit1_->num_rows());
+}
+
+TEST_F(TestCDCMultipleRowGroups, UpdateOnce) {
+  auto constexpr kRowGroupLength = 128 * 1024;
+  auto constexpr kEnableDictionary = false;
+  auto constexpr kMinChunkSize = 0 * 1024;
+  auto constexpr kMaxChunkSize = 128 * 1024;
+
+  ASSERT_OK_AND_ASSIGN(auto base,
+                       ConcatAndCombine({part1_, edit1_, part2_, part3_, edit2_}));
+  ASSERT_OK_AND_ASSIGN(auto updated,
+                       ConcatAndCombine({part1_, edit3_, part2_, part3_, edit2_}));
+  ASSERT_FALSE(base->Equals(*updated));
+
+  ASSERT_OK_AND_ASSIGN(
+      auto base_info,
+      WriteAndGetParquetInfo(base, kMinChunkSize, kMaxChunkSize, kEnableDictionary,
+                             ParquetDataPageVersion::V1, kRowGroupLength));
+  ASSERT_OK_AND_ASSIGN(
+      auto updated_info,
+      WriteAndGetParquetInfo(updated, kMinChunkSize, kMaxChunkSize, kEnableDictionary,
+                             ParquetDataPageVersion::V1, kRowGroupLength));
+
+  ASSERT_EQ(base_info.size(), 7);
+  ASSERT_EQ(updated_info.size(), 7);
+
+  ASSERT_EQ(base_info.at(0).page_lengths, updated_info.at(0).page_lengths);
+  ASSERT_EQ(base_info.at(1).page_lengths, updated_info.at(1).page_lengths);
+  AssertPageLengthDifferences(base_info.at(2), updated_info.at(2),
+                              /*max_number_of_equal_diffs=*/1);
+  for (size_t i = 2; i < updated_info.size(); i++) {
+    ASSERT_EQ(base_info.at(i).page_lengths, updated_info.at(i).page_lengths);
+  }

Review Comment:
   `AssertPageLengthDifferences` checks the page differences within a row group. We expect a single diff in the third row group updating values but not shifting any of the pages because the number of the values are the same (otherwise it would be an update + insertion/deletion). All the other row groups should be identical.
   
   Added a comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2027908418


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int8_t norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes
+  int64_t target_size = avg_chunk_size - min_chunk_size;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // 3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  if (norm_factor < -3 || norm_factor > 3) {
+    ARROW_LOG(WARNING) << "norm_factor=" << norm_factor
+                       << " is outside the recommended range [-3, 3]";
+  }
+
+  int mask_adjustement = 3 + norm_factor;
+  int effective_bits = mask_bits - mask_adjustement;
+  if (effective_bits == 0) {
+    return 0;
+  } else if (effective_bits > 64) {
+    throw ParquetException("The number of bits in the mask cannot exceed 64, got " +
+                           std::to_string(effective_bits));
+  } else if (effective_bits < 0) {
+    throw ParquetException(
+        "The difference between min_chunk_size=" + std::to_string(min_chunk_size) +
+        " and max_chunk_size=" + std::to_string(max_chunk_size) +
+        " is too small for the given norm_factor=" + std::to_string(norm_factor) +
+        ", either increase the size range or decrease the norm_factor.");
+  } else {
+    // create the mask by setting the top mask_bits bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int8_t norm_factor)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_factor)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(const bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+    has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+  }
+
+  template <int ByteWidth>
+  void Roll(const uint8_t* value) {
+    // Update the rolling hash with a compile-time known sized value, set has_matched_ to
+    // true if the hash matches the mask.
+
+    chunk_size_ += ByteWidth;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (size_t i = 0; i < ByteWidth; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  template <typename T>
+  void Roll(const T* value) {
+    return Roll<sizeof(T)>(reinterpret_cast<const uint8_t*>(value));
+  }
+
+  void Roll(const uint8_t* value, int64_t length) {
+    // Update the rolling hash with a binary-like value, set has_matched_ to true if the
+    // hash matches the mask.
+
+    chunk_size_ += length;
+    if (chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    for (auto i = 0; i < length; ++i) {
+      rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value[i]];
+      has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+    }
+  }
+
+  bool NeedNewChunk() {
+    // decide whether to create a new chunk based on the rolling hash; has_matched_ is
+    // set to true if we encountered a match since the last NeedNewChunk() call
+    if (ARROW_PREDICT_FALSE(has_matched_)) {
+      has_matched_ = false;
+      // in order to have a normal distribution of chunk sizes, we only create a new chunk
+      // if the adjused mask matches the rolling hash 8 times in a row, each run uses a
+      // different gearhash table (gearhash's chunk size has geometric distribution, and
+      // we use central limit theorem to approximate normal distribution, see
+      // section 6.2.1 in paper https://www.cidrdb.org/cidr2023/papers/p43-low.pdf)
+      if (ARROW_PREDICT_FALSE(++nth_run_ >= 7)) {
+        nth_run_ = 0;
+        chunk_size_ = 0;
+        return true;
+      }
+    }
+    if (ARROW_PREDICT_FALSE(chunk_size_ >= max_chunk_size_)) {
+      // we have a hard limit on the maximum chunk size, note that we don't reset the
+      // rolling hash state here, so the next NeedNewChunk() call will continue from the
+      // current state
+      chunk_size_ = 0;
+      return true;
+    }
+    return false;
+  }
+
+  void ValidateChunks(const std::vector<Chunk>& chunks, int64_t num_levels) {
+    // chunks must be non-empty and monotonic increasing
+    DCHECK(!chunks.empty());
+
+    // the first chunk must start at the first level
+    auto first_chunk = chunks.front();
+    DCHECK_EQ(first_chunk.level_offset, 0);
+    DCHECK_EQ(first_chunk.value_offset, 0);
+
+    // the following chunks must be contiguous, non-overlapping and monotonically
+    // increasing
+    auto sum_levels = first_chunk.levels_to_write;
+    for (size_t i = 1; i < chunks.size(); ++i) {
+      auto chunk = chunks[i];
+      auto prev_chunk = chunks[i - 1];
+      DCHECK_GT(chunk.levels_to_write, 0);
+      DCHECK_GE(chunk.value_offset, prev_chunk.value_offset);
+      DCHECK_EQ(chunk.level_offset, prev_chunk.level_offset + prev_chunk.levels_to_write);
+      sum_levels += chunk.levels_to_write;
+    }
+    DCHECK_EQ(sum_levels, num_levels);
+
+    // the last chunk must end at the last level
+    auto last_chunk = chunks.back();
+    DCHECK_EQ(last_chunk.level_offset + last_chunk.levels_to_write, num_levels);
+  }
+
+  template <typename RollFunc>
+  std::vector<Chunk> Calculate(const int16_t* def_levels, const int16_t* rep_levels,
+                               int64_t num_levels, const RollFunc& RollValue) {
+    // Calculate the chunk boundaries for typed Arrow arrays.
+    std::vector<Chunk> chunks;
+    int64_t offset;
+    int64_t prev_offset = 0;
+    int64_t prev_value_offset = 0;
+    bool has_def_levels = level_info_.def_level > 0;
+    bool has_rep_levels = level_info_.rep_level > 0;
+
+    if (!has_rep_levels && !has_def_levels) {
+      // fastest path for non-nested non-null data
+      for (offset = 0; offset < num_levels; ++offset) {
+        RollValue(offset);
+        if (NeedNewChunk()) {
+          chunks.push_back({prev_offset, prev_offset, offset - prev_offset});
+          prev_offset = offset;
+        }
+      }
+      // set the previous value offset to add the last chunk
+      prev_value_offset = prev_offset;
+    } else if (!has_rep_levels) {
+      // non-nested data with nulls
+      int16_t def_level;
+      for (int64_t offset = 0; offset < num_levels; ++offset) {
+        def_level = def_levels[offset];
+
+        Roll(&def_level);
+        if (def_level == level_info_.def_level) {
+          RollValue(offset);
+        }
+        if (NeedNewChunk()) {
+          chunks.push_back({prev_offset, prev_offset, offset - prev_offset});
+          prev_offset = offset;
+        }
+      }
+      // set the previous value offset to add the last chunk
+      prev_value_offset = prev_offset;
+    } else {
+      // nested data with nulls
+      int16_t def_level;
+      int16_t rep_level;
+      int64_t value_offset = 0;
+
+      for (offset = 0; offset < num_levels; ++offset) {
+        def_level = def_levels[offset];
+        rep_level = rep_levels[offset];
+
+        Roll(&def_level);
+        Roll(&rep_level);
+        if (def_level == level_info_.def_level) {
+          RollValue(value_offset);
+        }
+
+        if ((rep_level == 0) && NeedNewChunk()) {
+          // if we are at a record boundary and need a new chunk, we create a new chunk
+          auto levels_to_write = offset - prev_offset;
+          if (levels_to_write > 0) {
+            chunks.push_back({prev_offset, prev_value_offset, levels_to_write});
+            prev_offset = offset;
+            prev_value_offset = value_offset;
+          }
+        }
+        if (def_level >= level_info_.repeated_ancestor_def_level) {
+          // we only increment the value offset if we have a leaf value
+          ++value_offset;
+        }
+      }
+    }
+
+    // add the last chunk if we have any levels left
+    if (prev_offset < num_levels) {
+      chunks.push_back({prev_offset, prev_value_offset, num_levels - prev_offset});
+    }
+#ifndef NDEBUG
+    ValidateChunks(chunks, num_levels);
+#endif
+
+    return chunks;
+  }
+
+  template <int kByteWidth>
+  std::vector<Chunk> CalculateFixedWidth(const int16_t* def_levels,
+                                         const int16_t* rep_levels, int64_t num_levels,
+                                         const ::arrow::Array& values) {
+    const uint8_t* raw_values =
+        values.data()->GetValues<uint8_t>(1, 0) + values.offset() * kByteWidth;
+    return Calculate(def_levels, rep_levels, num_levels, [&](int64_t i) {
+      return Roll<kByteWidth>(&raw_values[i * kByteWidth]);
+    });
+  }
+
+  template <typename ArrayType>
+  std::vector<Chunk> CalculateBinaryLike(const int16_t* def_levels,
+                                         const int16_t* rep_levels, int64_t num_levels,
+                                         const ::arrow::Array& values) {
+    const auto& array = checked_cast<const ArrayType&>(values);
+    const uint8_t* value;
+    typename ArrayType::offset_type length;
+    return Calculate(def_levels, rep_levels, num_levels, [&](int64_t i) {
+      value = array.GetValue(i, &length);
+      Roll(value, length);
+    });

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025069939


##########
cpp/src/parquet/chunker_internal.cc:
##########
@@ -0,0 +1,413 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/chunker_internal.h"
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/util/bit_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/visit_type_inline.h"
+#include "parquet/chunker_internal_generated.h"
+#include "parquet/exception.h"
+#include "parquet/level_conversion.h"
+
+namespace parquet::internal {
+
+using ::arrow::internal::checked_cast;
+
+/// Calculate the mask to use for the rolling hash, the mask is used to determine if a
+/// new chunk should be created based on the rolling hash value. The mask is calculated
+/// based on the min_chunk_size, max_chunk_size and norm_factor parameters.
+///
+/// Assuming that the gear hash hash random values with a uniform distribution, then each
+/// bit in the actual value of rolling_hash_ has even probability of being set so a mask
+/// with the top N bits set has a probability of 1/2^N of matching the rolling hash. This
+/// is the judgment criteria for the original gear hash based content-defined chunking.
+/// The main drawback of this approach is the non-uniform distribution of the chunk sizes.
+///
+/// Later on the FastCDC has improved the process by introducing:
+/// - sub-minimum chunk cut-point skipping (not hashing the first `min_chunk_size` bytes)
+/// - chunk size normalization (using two masks)
+///
+/// This implementation uses cut-point skipping because it improves the overall
+/// performance and a more accurate alternative to have less skewed chunk size
+/// distribution. Instead of using two different masks (one with a lower and one with a
+/// higher probability of matching and switching them based on the actual chunk size), we
+/// rather use 8 different gear hash tables and require having 8 consecutive matches while
+/// switching between the used hashtables. This approach is based on central limit theorem
+/// and approximates normal distribution of the chunk sizes.
+//
+// @param min_chunk_size The minimum chunk size (default 256KiB)
+// @param max_chunk_size The maximum chunk size (default 1MiB)
+// @param norm_factor Normalization factor (default 0)
+// @return The mask used to compare against the rolling hash
+static uint64_t CalculateMask(int64_t min_chunk_size, int64_t max_chunk_size,
+                              int8_t norm_factor) {
+  if (min_chunk_size < 0) {
+    throw ParquetException("min_chunk_size must be positive");
+  }
+  if (max_chunk_size <= min_chunk_size) {
+    throw ParquetException("max_chunk_size must be greater than min_chunk_size");
+  }
+
+  // calculate the average size of the chunks
+  int64_t avg_chunk_size = (min_chunk_size + max_chunk_size) / 2;
+  // since we are skipping the first `min_chunk_size` bytes for each chunk, we need to
+  // target a smaller chunk size to reach the average size after skipping the first
+  // `min_chunk_size` bytes
+  int64_t target_size = avg_chunk_size - min_chunk_size;
+
+  // assuming that the gear hash has a uniform distribution, we can calculate the mask
+  // by taking the floor(log2(target_size))
+  int mask_bits = std::max(0, ::arrow::bit_util::NumRequiredBits(target_size) - 1);
+
+  // 3 because we are using 8 hash tables to have more gaussian-like distribution,
+  // a user defined `norm_factor` can be used to adjust the mask size, hence the matching
+  // probability, by increasing the norm_factor we increase the probability of matching
+  // the mask, forcing the distribution closer to the average size; norm_factor is 0 by
+  // default
+  if (norm_factor < -3 || norm_factor > 3) {
+    ARROW_LOG(WARNING) << "norm_factor=" << norm_factor
+                       << " is outside the recommended range [-3, 3]";
+  }
+
+  int mask_adjustement = 3 + norm_factor;
+  int effective_bits = mask_bits - mask_adjustement;
+  if (effective_bits == 0) {
+    return 0;
+  } else if (effective_bits > 64) {
+    throw ParquetException("The number of bits in the mask cannot exceed 64, got " +
+                           std::to_string(effective_bits));
+  } else if (effective_bits < 0) {
+    throw ParquetException(
+        "The difference between min_chunk_size=" + std::to_string(min_chunk_size) +
+        " and max_chunk_size=" + std::to_string(max_chunk_size) +
+        " is too small for the given norm_factor=" + std::to_string(norm_factor) +
+        ", either increase the size range or decrease the norm_factor.");
+  } else {
+    // create the mask by setting the top mask_bits bits
+    return std::numeric_limits<uint64_t>::max() << (64 - effective_bits);
+  }
+}
+
+class ContentDefinedChunker::Impl {
+ public:
+  Impl(const LevelInfo& level_info, int64_t min_chunk_size, int64_t max_chunk_size,
+       int8_t norm_factor)
+      : level_info_(level_info),
+        min_chunk_size_(min_chunk_size),
+        max_chunk_size_(max_chunk_size),
+        rolling_hash_mask_(CalculateMask(min_chunk_size, max_chunk_size, norm_factor)) {}
+
+  uint64_t GetRollingHashMask() const { return rolling_hash_mask_; }
+
+  void Roll(const bool value) {
+    if (++chunk_size_ < min_chunk_size_) {
+      // short-circuit if we haven't reached the minimum chunk size, this speeds up the
+      // chunking process since the gearhash doesn't need to be updated
+      return;
+    }
+    rolling_hash_ = (rolling_hash_ << 1) + kGearhashTable[nth_run_][value];
+    has_matched_ = has_matched_ || ((rolling_hash_ & rolling_hash_mask_) == 0);
+  }
+
+  template <int ByteWidth>

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025208441


##########
cpp/src/parquet/chunker_internal_test.cc:
##########
@@ -0,0 +1,1397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+#include <algorithm>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/table.h"
+#include "arrow/testing/extension_type.h"
+#include "arrow/testing/generator.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/float16.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/reader_internal.h"
+#include "parquet/arrow/test_util.h"
+#include "parquet/arrow/writer.h"
+#include "parquet/chunker_internal.h"
+#include "parquet/column_writer.h"
+#include "parquet/file_writer.h"
+
+namespace parquet::internal {
+
+using ::arrow::Array;
+using ::arrow::ChunkedArray;
+using ::arrow::ConcatenateTables;
+using ::arrow::DataType;
+using ::arrow::default_memory_pool;
+using ::arrow::Field;
+using ::arrow::Result;
+using ::arrow::Schema;
+using ::arrow::Table;
+using ::arrow::io::BufferReader;
+using ::parquet::arrow::FileReader;
+using ::parquet::arrow::FileReaderBuilder;
+using ::parquet::arrow::MakeSimpleTable;
+using ::parquet::arrow::NonNullArray;
+using ::parquet::arrow::WriteTable;
+
+using ::testing::Bool;
+using ::testing::Combine;
+using ::testing::Values;
+
+// generate determinisic and platform-independent data
+inline uint64_t hash(uint64_t seed, uint64_t index) {
+  uint64_t h = (index + seed) * 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  h *= 0xff51afd7ed558ccdull;
+  h ^= h >> 33;
+  h *= 0xc4ceb9fe1a85ec53ull;
+  h ^= h >> 33;
+  return h;
+}
+
+template <typename BuilderType, typename ValueFunc>
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<DataType>& type,
+                                             bool nullable, int64_t length, uint64_t seed,
+                                             ValueFunc value_func) {
+  BuilderType builder(type, default_memory_pool());
+
+  if (nullable) {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      if (val % 10 == 0) {
+        RETURN_NOT_OK(builder.AppendNull());
+      } else {
+        RETURN_NOT_OK(builder.Append(value_func(val)));
+      }
+    }
+  } else {
+    for (int64_t i = 0; i < length; ++i) {
+      uint64_t val = hash(seed, i);
+      RETURN_NOT_OK(builder.Append(value_func(val)));
+    }
+  }
+
+  std::shared_ptr<Array> array;
+  RETURN_NOT_OK(builder.Finish(&array));
+  RETURN_NOT_OK(array->ValidateFull());
+  return array;
+}
+
+#define GENERATE_CASE(TYPE_ID, BUILDER_TYPE, VALUE_EXPR)                          \
+  case ::arrow::Type::TYPE_ID: {                                                  \
+    auto value_func = [](uint64_t val) { return VALUE_EXPR; };                    \
+    return GenerateArray<BUILDER_TYPE>(type, nullable, length, seed, value_func); \
+  }
+
+Result<std::shared_ptr<Array>> GenerateArray(const std::shared_ptr<Field>& field,
+                                             int64_t length, int64_t seed) {
+  const std::shared_ptr<DataType>& type = field->type();
+  bool nullable = field->nullable();
+
+  switch (type->id()) {
+    GENERATE_CASE(BOOL, ::arrow::BooleanBuilder, (val % 2 == 0))
+
+    // Numeric types.
+    GENERATE_CASE(INT8, ::arrow::Int8Builder, static_cast<int8_t>(val))
+    GENERATE_CASE(INT16, ::arrow::Int16Builder, static_cast<int16_t>(val))
+    GENERATE_CASE(INT32, ::arrow::Int32Builder, static_cast<int32_t>(val))
+    GENERATE_CASE(INT64, ::arrow::Int64Builder, static_cast<int64_t>(val))
+    GENERATE_CASE(UINT8, ::arrow::UInt8Builder, static_cast<uint8_t>(val))
+    GENERATE_CASE(UINT16, ::arrow::UInt16Builder, static_cast<uint16_t>(val))
+    GENERATE_CASE(UINT32, ::arrow::UInt32Builder, static_cast<uint32_t>(val))
+    GENERATE_CASE(UINT64, ::arrow::UInt64Builder, static_cast<uint64_t>(val))
+    GENERATE_CASE(HALF_FLOAT, ::arrow::HalfFloatBuilder,
+                  static_cast<uint16_t>(val % 1000))
+    GENERATE_CASE(FLOAT, ::arrow::FloatBuilder, static_cast<float>(val % 1000) / 1000.0f)
+    GENERATE_CASE(DOUBLE, ::arrow::DoubleBuilder,
+                  static_cast<double>(val % 100000) / 1000.0)
+    case ::arrow::Type::DECIMAL128: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal128Type&>(*type);
+      // Limit the value to fit within the specified precision
+      int32_t max_exponent = decimal_type.precision() - decimal_type.scale();
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal128(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal128Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+    case ::arrow::Type::DECIMAL256: {
+      const auto& decimal_type = static_cast<const ::arrow::Decimal256Type&>(*type);
+      // Limit the value to fit within the specified precision, capped at 9 to avoid
+      // int64_t overflow
+      int32_t max_exponent = std::min(9, decimal_type.precision() - decimal_type.scale());
+      int64_t max_value = static_cast<int64_t>(std::pow(10, max_exponent) - 1);
+      auto value_func = [&](uint64_t val) {
+        return ::arrow::Decimal256(val % max_value);
+      };
+      return GenerateArray<::arrow::Decimal256Builder>(type, nullable, length, seed,
+                                                       value_func);
+    }
+
+      // Temporal types
+      GENERATE_CASE(DATE32, ::arrow::Date32Builder, static_cast<int32_t>(val))
+      GENERATE_CASE(TIME32, ::arrow::Time32Builder,
+                    std::abs(static_cast<int32_t>(val) % 86400000))
+      GENERATE_CASE(TIME64, ::arrow::Time64Builder,
+                    std::abs(static_cast<int64_t>(val) % 86400000000))
+      GENERATE_CASE(TIMESTAMP, ::arrow::TimestampBuilder, static_cast<int64_t>(val))
+      GENERATE_CASE(DURATION, ::arrow::DurationBuilder, static_cast<int64_t>(val))
+
+      // Binary and string types.
+      GENERATE_CASE(STRING, ::arrow::StringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(LARGE_STRING, ::arrow::LargeStringBuilder,
+                    std::string("str_") + std::to_string(val))
+      GENERATE_CASE(BINARY, ::arrow::BinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+      GENERATE_CASE(LARGE_BINARY, ::arrow::LargeBinaryBuilder,
+                    std::string("bin_") + std::to_string(val))
+    case ::arrow::Type::FIXED_SIZE_BINARY: {
+      auto size = static_cast<::arrow::FixedSizeBinaryType*>(type.get())->byte_width();
+      auto value_func = [size](uint64_t val) {
+        return std::string("bin_") + std::to_string(val).substr(0, size - 4);
+      };
+      return GenerateArray<::arrow::FixedSizeBinaryBuilder>(type, nullable, length, seed,
+                                                            value_func);
+    }
+
+    case ::arrow::Type::STRUCT: {
+      auto struct_type = static_cast<::arrow::StructType*>(type.get());
+      std::vector<std::shared_ptr<Array>> child_arrays;
+      for (auto i = 0; i < struct_type->num_fields(); i++) {
+        ARROW_ASSIGN_OR_RAISE(auto child_array,
+                              GenerateArray(struct_type->field(i), length,
+                                            seed + static_cast<uint64_t>(i + 300)));
+        child_arrays.push_back(child_array);
+      }
+      auto struct_array =
+          std::make_shared<::arrow::StructArray>(type, length, child_arrays);
+      return struct_array;
+    }
+
+    case ::arrow::Type::LIST: {
+      auto list_type = static_cast<::arrow::ListType*>(type.get());
+      auto value_field = ::arrow::field("item", list_type->value_type());
+      ARROW_ASSIGN_OR_RAISE(auto values_array, GenerateArray(value_field, length, seed));
+      auto offset_builder = ::arrow::Int32Builder();
+      auto bitmap_builder = ::arrow::TypedBufferBuilder<bool>();
+
+      int32_t num_nulls = 0;
+      int32_t num_elements = 0;
+      uint8_t element_size = 0;
+      int32_t current_offset = 0;
+      RETURN_NOT_OK(offset_builder.Append(current_offset));
+      while (current_offset < length) {
+        num_elements++;
+        auto is_valid = !(nullable && (num_elements % 10 == 0));
+        if (is_valid) {
+          RETURN_NOT_OK(bitmap_builder.Append(true));
+          current_offset += element_size;
+          if (current_offset > length) {
+            RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(length)));
+            break;
+          } else {
+            RETURN_NOT_OK(offset_builder.Append(current_offset));
+          }
+        } else {
+          RETURN_NOT_OK(offset_builder.Append(static_cast<int32_t>(current_offset)));
+          RETURN_NOT_OK(bitmap_builder.Append(false));
+          num_nulls++;
+        }
+
+        if (element_size > 4) {
+          element_size = 0;
+        } else {
+          element_size++;
+        }
+      }
+
+      std::shared_ptr<Array> offsets_array;
+      RETURN_NOT_OK(offset_builder.Finish(&offsets_array));
+      std::shared_ptr<Buffer> bitmap_buffer;
+      RETURN_NOT_OK(bitmap_builder.Finish(&bitmap_buffer));
+      ARROW_ASSIGN_OR_RAISE(
+          auto list_array, ::arrow::ListArray::FromArrays(
+                               type, *offsets_array, *values_array, default_memory_pool(),
+                               bitmap_buffer, num_nulls));
+      RETURN_NOT_OK(list_array->ValidateFull());
+      return list_array;
+    }
+
+    case ::arrow::Type::EXTENSION: {
+      auto extension_type = dynamic_cast<::arrow::ExtensionType*>(type.get());
+      auto storage_type = extension_type->storage_type();
+      auto storage_field = ::arrow::field("storage", storage_type, true);
+      ARROW_ASSIGN_OR_RAISE(auto storage_array,
+                            GenerateArray(storage_field, length, seed));
+      return ::arrow::ExtensionType::WrapArray(type, storage_array);
+    }
+
+    default:
+      return ::arrow::Status::NotImplemented("Unsupported data type " + type->ToString());
+  }
+}
+
+Result<std::shared_ptr<Table>> GenerateTable(
+    const std::shared_ptr<::arrow::Schema>& schema, int64_t size, int64_t seed = 0) {
+  std::vector<std::shared_ptr<Array>> arrays;
+  for (const auto& field : schema->fields()) {
+    ARROW_ASSIGN_OR_RAISE(auto array, GenerateArray(field, size, seed));

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-45750: [C++][Python][Parquet] Implement Content-Defined Chunking for the Parquet writer [arrow]

["kszucs (via GitHub)" <gi...@apache.org>]

kszucs commented on code in PR #45360:
URL: https://github.com/apache/arrow/pull/45360#discussion_r2025098659


##########
cpp/src/parquet/properties.h:
##########
@@ -245,6 +245,14 @@ class PARQUET_EXPORT ColumnProperties {
   bool page_index_enabled_;
 };
 
+struct CDCOptions {
+  int64_t min_chunk_size;
+  int64_t max_chunk_size;
+  int8_t norm_factor;
+};
+
+static constexpr CDCOptions kDefaultCdcOptions = CDCOptions{256 * 1024, 1024 * 1024, 0};

Review Comment:
   Regarding the data page size, the default is
   ```cpp
   static constexpr int64_t kDefaultDataPageSize = 1024 * 1024;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org