You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/30 06:35:06 UTC
(datafusion-comet) 01/01: Publish built docs triggered by dd7d355fc3e72837b394345fe08b8677d9c9090b
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 5c7d480c26dbb54d854f34641dfc784d1d36e38a
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Wed Sep 30 06:34:59 2026 +0000
Publish built docs triggered by dd7d355fc3e72837b394345fe08b8677d9c9090b
---
...id-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg | 2 +-
_sources/contributor-guide/iceberg-writes.md.txt | 21 +++++++++++++++-----
contributor-guide/iceberg-writes.html | 23 ++++++++++++++++------
searchindex.js | 2 +-
4 files changed, 35 insertions(+), 13 deletions(-)
diff --git a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
index 109cbb7664..e0d1fa6560 100644
--- a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
+++ b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/iceberg-writes.md.txt b/_sources/contributor-guide/iceberg-writes.md.txt
index 63ec6ea5d7..b30720f9e9 100644
--- a/_sources/contributor-guide/iceberg-writes.md.txt
+++ b/_sources/contributor-guide/iceberg-writes.md.txt
@@ -209,10 +209,17 @@ Points where Comet adapts iceberg-rust to match iceberg-java:
- **Location generation.** `CometLocationGenerator` (`iceberg_partition_path.rs`) replaces
iceberg-rust's `DefaultLocationGenerator` and renders partition directories the way iceberg-java's
`PartitionSpec.partitionToPath` does. iceberg-rust's own rendering differs for several types, and
- panics for pre-1970 `timestamptz`. Float and double values use Comet's Java `Double.toString`
- renderer (`java_float_string`), shared with `cast(float as string)`. The partition type is
- resolved once when the generator is built, because `LocationGenerator::generate_location` cannot
- return an error.
+ panics for pre-1970 `timestamptz` and for dates past `chrono`'s calendar. Float and double values
+ use Comet's Java `Double.toString` renderer (`java_float_string`), shared with
+ `cast(float as string)`. The partition type is resolved once when the generator is built, because
+ `LocationGenerator::generate_location` cannot return an error.
+- **Partition values.** `PartitionValueCalculator` (`iceberg_partition_value.rs`) replaces
+ iceberg-rust's calculator of the same name, and `PartitionSplitter` replaces its
+ `RecordBatchPartitionSplitter`. `year` and `month` go through Comet's `iceberg_years` /
+ `iceberg_months` kernels, the ones the sort in front of a clustered write runs. iceberg-rust
+ computes them with Arrow's `date_part`, which returns NULL past `chrono`'s calendar
+ ([#6145](https://github.com/apache/datafusion-comet/issues/6145)). Every other transform stays on
+ iceberg-rust.
- **File names.** `file_name_prefix` embeds the partition id, the task attempt id and the operation
id, so a retried or speculative attempt never reuses another attempt's file names.
- **Row pacing.** iceberg-java's rolling writer checks the target file size every 1000 rows of the
@@ -323,7 +330,7 @@ the writer: run the write suites and the Iceberg Spark tests. The pin policy is
| `CometIcebergSystemFunctionSuite` | Native `bucket`, `truncate`, `years`/`months`/`days`/`hours`, which keep a partitioned write's hash distribution and sort native end to end. |
| `CometIcebergRewriteActionSuite` | Iceberg's `rewrite_data_files` with the split plan and the native writer. |
| `IcebergWriteProtoTranslationSuite` | Translation of properties into `IcebergParquetWriteSettings` and the writer mode. |
-| Rust tests in `iceberg_write.rs` and `iceberg_partition_path.rs` | File rolling on the 1000-row grid, fanout order, clustered input checks, cleanup guard, manifest round trip, partition path rendering. |
+| Rust tests in `iceberg_write.rs` and in `iceberg_partition_*.rs` | File rolling on the 1000-row grid, fanout order, clustered input checks, cleanup guard, manifest round trip, partition path rendering, partition values past `chrono`'s calendar. |
| `CometIcebergWriteBenchmark` | Native versus iceberg-java for unpartitioned, clustered, fanout and copy-on-write delete writes. It checks each arm's plan before timing it. |
The Comet suites run against the Iceberg version each Spark profile pins in `spark/pom.xml`: 1.5.2
@@ -368,6 +375,10 @@ Each of these has caused a bug on this path:
- **Map iteration order leaks.** It reaches file names, manifest order and read order.
- **Values that compare equal in Rust may not in Java.** Signed zeros and NaN under `OrderedFloat`,
and string or float rendering in partition paths.
+- **`chrono` stops at year 262142.** A Spark date reaches year 5881580 and a timestamp year 294247,
+ and iceberg-java handles all of them. Code that goes through `chrono`, including Arrow's
+ `date_part`, panics or returns NULL past that
+ ([#6145](https://github.com/apache/datafusion-comet/issues/6145)).
- **Partition evolution leaves `void` fields behind.** A v1 spec keeps a dropped partition field as
a `void` transform, whose source column may later be dropped from the schema. Resolving the spec
against the schema then fails
diff --git a/contributor-guide/iceberg-writes.html b/contributor-guide/iceberg-writes.html
index f42c52ef97..30e48e8d9d 100644
--- a/contributor-guide/iceberg-writes.html
+++ b/contributor-guide/iceberg-writes.html
@@ -761,10 +761,17 @@ task:</p>
<li><p><strong>Location generation.</strong> <code class="docutils literal notranslate"><span class="pre">CometLocationGenerator</span></code> (<code class="docutils literal notranslate"><span class="pre">iceberg_partition_path.rs</span></code>) replaces
iceberg-rust’s <code class="docutils literal notranslate"><span class="pre">DefaultLocationGenerator</span></code> and renders partition directories the way iceberg-java’s
<code class="docutils literal notranslate"><span class="pre">PartitionSpec.partitionToPath</span></code> does. iceberg-rust’s own rendering differs for several types, and
-panics for pre-1970 <code class="docutils literal notranslate"><span class="pre">timestamptz</span></code>. Float and double values use Comet’s Java <code class="docutils literal notranslate"><span class="pre">Double.toString</span></code>
-renderer (<code class="docutils literal notranslate"><span class="pre">java_float_string</span></code>), shared with <code class="docutils literal notranslate"><span class="pre">cast(float</span> <span class="pre">as</span> <span class="pre">string)</span></code>. The partition type is
-resolved once when the generator is built, because <code class="docutils literal notranslate"><span class="pre">LocationGenerator::generate_location</span></code> cannot
-return an error.</p></li>
+panics for pre-1970 <code class="docutils literal notranslate"><span class="pre">timestamptz</span></code> and for dates past <code class="docutils literal notranslate"><span class="pre">chrono</span></code>’s calendar. Float and double values
+use Comet’s Java <code class="docutils literal notranslate"><span class="pre">Double.toString</span></code> renderer (<code class="docutils literal notranslate"><span class="pre">java_float_string</span></code>), shared with
+<code class="docutils literal notranslate"><span class="pre">cast(float</span> <span class="pre">as</span> <span class="pre">string)</span></code>. The partition type is resolved once when the generator is built, because
+<code class="docutils literal notranslate"><span class="pre">LocationGenerator::generate_location</span></code> cannot return an error.</p></li>
+<li><p><strong>Partition values.</strong> <code class="docutils literal notranslate"><span class="pre">PartitionValueCalculator</span></code> (<code class="docutils literal notranslate"><span class="pre">iceberg_partition_value.rs</span></code>) replaces
+iceberg-rust’s calculator of the same name, and <code class="docutils literal notranslate"><span class="pre">PartitionSplitter</span></code> replaces its
+<code class="docutils literal notranslate"><span class="pre">RecordBatchPartitionSplitter</span></code>. <code class="docutils literal notranslate"><span class="pre">year</span></code> and <code class="docutils literal notranslate"><span class="pre">month</span></code> go through Comet’s <code class="docutils literal notranslate"><span class="pre">iceberg_years</span></code> /
+<code class="docutils literal notranslate"><span class="pre">iceberg_months</span></code> kernels, the ones the sort in front of a clustered write runs. iceberg-rust
+computes them with Arrow’s <code class="docutils literal notranslate"><span class="pre">date_part</span></code>, which returns NULL past <code class="docutils literal notranslate"><span class="pre">chrono</span></code>’s calendar
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6145">#6145</a>). Every other transform stays on
+iceberg-rust.</p></li>
<li><p><strong>File names.</strong> <code class="docutils literal notranslate"><span class="pre">file_name_prefix</span></code> embeds the partition id, the task attempt id and the operation
id, so a retried or speculative attempt never reuses another attempt’s file names.</p></li>
<li><p><strong>Row pacing.</strong> iceberg-java’s rolling writer checks the target file size every 1000 rows of the
@@ -909,8 +916,8 @@ the writer: run the write suites and the Iceberg Spark tests. The pin policy is
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">IcebergWriteProtoTranslationSuite</span></code></p></td>
<td><p>Translation of properties into <code class="docutils literal notranslate"><span class="pre">IcebergParquetWriteSettings</span></code> and the writer mode.</p></td>
</tr>
-<tr class="row-odd"><td><p>Rust tests in <code class="docutils literal notranslate"><span class="pre">iceberg_write.rs</span></code> and <code class="docutils literal notranslate"><span class="pre">iceberg_partition_path.rs</span></code></p></td>
-<td><p>File rolling on the 1000-row grid, fanout order, clustered input checks, cleanup guard, manifest round trip, partition path rendering.</p></td>
+<tr class="row-odd"><td><p>Rust tests in <code class="docutils literal notranslate"><span class="pre">iceberg_write.rs</span></code> and in <code class="docutils literal notranslate"><span class="pre">iceberg_partition_*.rs</span></code></p></td>
+<td><p>File rolling on the 1000-row grid, fanout order, clustered input checks, cleanup guard, manifest round trip, partition path rendering, partition values past <code class="docutils literal notranslate"><span class="pre">chrono</span></code>’s calendar.</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">CometIcebergWriteBenchmark</span></code></p></td>
<td><p>Native versus iceberg-java for unpartitioned, clustered, fanout and copy-on-write delete writes. It checks each arm’s plan before timing it.</p></td>
@@ -959,6 +966,10 @@ namespace that reaches the writer, as the <code class="docutils literal notransl
<li><p><strong>Map iteration order leaks.</strong> It reaches file names, manifest order and read order.</p></li>
<li><p><strong>Values that compare equal in Rust may not in Java.</strong> Signed zeros and NaN under <code class="docutils literal notranslate"><span class="pre">OrderedFloat</span></code>,
and string or float rendering in partition paths.</p></li>
+<li><p><strong><code class="docutils literal notranslate"><span class="pre">chrono</span></code> stops at year 262142.</strong> A Spark date reaches year 5881580 and a timestamp year 294247,
+and iceberg-java handles all of them. Code that goes through <code class="docutils literal notranslate"><span class="pre">chrono</span></code>, including Arrow’s
+<code class="docutils literal notranslate"><span class="pre">date_part</span></code>, panics or returns NULL past that
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6145">#6145</a>).</p></li>
<li><p><strong>Partition evolution leaves <code class="docutils literal notranslate"><span class="pre">void</span></code> fields behind.</strong> A v1 spec keeps a dropped partition field as
a <code class="docutils literal notranslate"><span class="pre">void</span></code> transform, whose source column may later be dropped from the schema. Resolving the spec
against the schema then fails
diff --git a/searchindex.js b/searchindex.js
index c77d7252e1..995f85c9b6 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org