You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "Amer-Mukhtar (via GitHub)" <gi...@apache.org> on 2026/07/23 10:13:38 UTC
[PR] test(pyamber): add unit tests for attribute_type_utils [texera]
Amer-Mukhtar opened a new pull request, #6833:
URL: https://github.com/apache/texera/pull/6833
Description
What changes were proposed in this PR?
- Add `amber/src/test/python/core/models/schema/test_attribute_type_utils.py`, a new pytest spec for the Python worker's `attribute_type_utils`, which previously had no dedicated unit tests.
- Cover `detect_attribute_type_from_arrow_field`: every supported Arrow type, timestamps across all four units, the `texera_type: LARGE_BINARY` metadata marker taking precedence over the Arrow type, metadata that doesn't mark large binary falling back to the type mapping, and unmapped Arrow types raising `KeyError`.
- Cover `create_arrow_field_with_metadata`: every `AttributeType`, metadata attached only for `LARGE_BINARY`, attribute names preserved verbatim (empty, unicode, emoji, long), and non-`AttributeType` input raising `KeyError`.
- Cover round trips, including that stripping the metadata off a `LARGE_BINARY` field degrades it to `STRING`.
Any related issues, documentation, discussions?
Closes: #6635
How was this PR tested?
- Run: `cd amber && pytest src/test/python/core/models/schema/test_attribute_type_utils.py -v`, expect all 67 tests passing.
- Coverage of the module is 100% (12/12 statements) via `--cov=core.models.schema.attribute_type_utils`.
- Verified the tests catch regressions by temporarily breaking the module: ignoring metadata during detection fails 6 tests, attaching metadata to every field fails 15.
- Test-only change; no production code is modified.
Was this PR authored or co-authored using generative AI tooling?
Co-authored with Claude
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(pyamber): add unit tests for attribute_type_utils [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6833:
URL: https://github.com/apache/texera/pull/6833#issuecomment-5563766903
This pull request has been automatically marked as **stale** because it has
not had any activity for 30 days. It will be closed in **7 days** if no
further activity occurs.
If this PR is still relevant, please:
- Push new commits or rebase onto the latest `main`, or
- Comment with a status update, or
- Address any open review feedback.
To exempt this PR from auto-close, a maintainer can add the `pinned` label.
Thank you for your contribution to Apache Texera (incubating)!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(pyamber): add unit tests for attribute_type_utils [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6833:
URL: https://github.com/apache/texera/pull/6833#issuecomment-5057236112
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- No candidates found from `git blame` history.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(pyamber): add unit tests for attribute_type_utils [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #6833:
URL: https://github.com/apache/texera/pull/6833#issuecomment-5057424280
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/6833?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 76.28%. Comparing base ([`0ba5d1a`](https://app.codecov.io/gh/apache/texera/commit/0ba5d1a22a3baaa61a36d1bbb8c4573d40e89b5c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`72392be`](https://app.codecov.io/gh/apache/texera/commit/72392beae2420dfe06c5d5175b7545befca8c16d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:white_check_mark: All tests successful. No failed tests found.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #6833 +/- ##
=========================================
Coverage 76.28% 76.28%
Complexity 3452 3452
=========================================
Files 1161 1161
Lines 45916 45916
Branches 5099 5099
=========================================
Hits 35025 35025
Misses 9327 9327
Partials 1564 1564
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/6833/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/6833/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.00% <ø> (ø)` | | Carriedforward from [0ba5d1a](https://app.codecov.io/gh/apache/texera/commit/0ba5d1a22a3baaa61a36d1bbb8c4573d40e89b5c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/6833/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.76% <ø> (ø)` | | Carriedforward from [0ba5d1a](https://app.codecov.io/gh/apache/texera/commit/0ba5d1a22a3baaa61a36d1bbb8c4573d40e89b5c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [amber](https://app.codecov.io/gh/apache/texera/pull/6833/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `67.33% <ø> (ø)` | | Carriedforward from [0ba5d1a](https://app.codecov.io/gh/apache/texera/commit/0ba5d1a22a3baaa61a36d1bbb8c4573d40e89b5c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/6833/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `20.49% <ø> (ø)` | | Carriedforward from [0ba5d1a](https://app.codecov.io/gh/apache/texera/commit/0ba5d1a22a3baaa61a36d1bbb8c4573d40e89b5c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/6833/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `66.66% <ø> (ø)` | | Carriedforward from [0ba5d1a](https://app.codecov.io/gh/apache/texera/commit/0ba5d1a22a3baaa61a36d1bbb8c4573d40e89b5c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/6833/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `67.21% <ø> (ø)` | | Carriedforward from [0ba5d1a](https://app.codecov.io/gh/apache/texera/commit/0ba5d1a22a3baaa61a36d1bbb8c4573d40e89b5c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/6833/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.72% <ø> (ø)` | | Carriedforward from [0ba5d1a](https://app.codecov.io/gh/apache/texera/commit/0ba5d1a22a3baaa61a36d1bbb8c4573d40e89b5c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/6833/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.94% <ø> (ø)` | | Carriedforward from [0ba5d1a](https://app.codecov.io/gh/apache/texera/commit/0ba5d1a22a3baaa61a36d1bbb8c4573d40e89b5c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/6833/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `92.15% <ø> (ø)` | | Carriedforward from [0ba5d1a](https://app.codecov.io/gh/apache/texera/commit/0ba5d1a22a3baaa61a36d1bbb8c4573d40e89b5c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/6833/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.14% <ø> (ø)` | | Carriedforward from [0ba5d1a](https://app.codecov.io/gh/apache/texera/commit/0ba5d1a22a3baaa61a36d1bbb8c4573d40e89b5c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/6833?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(pyamber): add unit tests for attribute_type_utils [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6833:
URL: https://github.com/apache/texera/pull/6833#issuecomment-5057359291
<!-- texera-benchmarks-comment -->
## ⚠️ Benchmark changes need a look
🟢 2 better · 🔴 3 worse · ⚪ 10 noise (<±5%) · 0 without baseline
> Compared against main `0ba5d1a` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/29998491963)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| 🔴 | bs=10 sw=10 sl=64 | 434 | 0.265 | 22,816/28,699/28,699 us | 🟢 -17.9% / 🔴 +78.5% |
| 🔴 | bs=100 sw=10 sl=64 | 924 | 0.564 | 105,233/156,049/156,049 us | 🔴 +19.2% / 🔴 +45.3% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,101 | 0.672 | 914,839/958,871/958,871 us | ⚪ within ±5% / 🟢 -9.2% |
<details><summary>Baseline details</summary>
Latest main `0ba5d1a` from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 434 tuples/sec | 418 tuples/sec | 760.2 tuples/sec | +3.8% | -42.9% |
| bs=10 sw=10 sl=64 | MB/s | 0.265 MB/s | 0.255 MB/s | 0.464 MB/s | +3.9% | -42.9% |
| bs=10 sw=10 sl=64 | p50 | 22,816 us | 20,344 us | 12,781 us | +12.2% | +78.5% |
| bs=10 sw=10 sl=64 | p95 | 28,699 us | 34,938 us | 16,118 us | -17.9% | +78.1% |
| bs=10 sw=10 sl=64 | p99 | 28,699 us | 34,938 us | 19,074 us | -17.9% | +50.5% |
| bs=100 sw=10 sl=64 | throughput | 924 tuples/sec | 963 tuples/sec | 988.68 tuples/sec | -4.0% | -6.5% |
| bs=100 sw=10 sl=64 | MB/s | 0.564 MB/s | 0.588 MB/s | 0.603 MB/s | -4.1% | -6.5% |
| bs=100 sw=10 sl=64 | p50 | 105,233 us | 102,314 us | 101,448 us | +2.9% | +3.7% |
| bs=100 sw=10 sl=64 | p95 | 156,049 us | 130,861 us | 107,410 us | +19.2% | +45.3% |
| bs=100 sw=10 sl=64 | p99 | 156,049 us | 130,861 us | 120,258 us | +19.2% | +29.8% |
| bs=1000 sw=10 sl=64 | throughput | 1,101 tuples/sec | 1,082 tuples/sec | 1,023 tuples/sec | +1.8% | +7.6% |
| bs=1000 sw=10 sl=64 | MB/s | 0.672 MB/s | 0.66 MB/s | 0.624 MB/s | +1.8% | +7.6% |
| bs=1000 sw=10 sl=64 | p50 | 914,839 us | 926,537 us | 983,888 us | -1.3% | -7.0% |
| bs=1000 sw=10 sl=64 | p95 | 958,871 us | 978,638 us | 1,030,100 us | -2.0% | -6.9% |
| bs=1000 sw=10 sl=64 | p99 | 958,871 us | 978,638 us | 1,056,513 us | -2.0% | -9.2% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,460.91,200,128000,434,0.265,22816.13,28698.68,28698.68
1,100,10,64,20,2163.64,2000,1280000,924,0.564,105232.52,156049.35,156049.35
2,1000,10,64,20,18168.18,20000,12800000,1101,0.672,914839.37,958870.50,958870.50
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(pyamber): add unit tests for attribute_type_utils [texera]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #6833:
URL: https://github.com/apache/texera/pull/6833#discussion_r3642790902
##########
amber/src/test/python/core/models/schema/test_attribute_type_utils.py:
##########
@@ -0,0 +1,220 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pyarrow as pa
+import pytest
+
+from core.models.schema.attribute_type import AttributeType
+from core.models.schema.attribute_type_utils import (
+ LARGE_BINARY_METADATA_VALUE,
+ TEXERA_TYPE_METADATA_KEY,
+ create_arrow_field_with_metadata,
+ detect_attribute_type_from_arrow_field,
+)
+
+LARGE_BINARY_METADATA = {TEXERA_TYPE_METADATA_KEY: LARGE_BINARY_METADATA_VALUE}
+
+
+class TestDetectAttributeTypeFromArrowField:
+ @pytest.mark.parametrize(
+ "arrow_type, expected",
+ [
+ (pa.int32(), AttributeType.INT),
+ (pa.int64(), AttributeType.LONG),
+ (pa.string(), AttributeType.STRING),
+ (pa.large_string(), AttributeType.STRING),
+ (pa.float64(), AttributeType.DOUBLE),
+ (pa.bool_(), AttributeType.BOOL),
+ (pa.binary(), AttributeType.BINARY),
+ (pa.large_binary(), AttributeType.BINARY),
+ (pa.timestamp("us"), AttributeType.TIMESTAMP),
+ ],
+ )
+ def test_detects_each_supported_arrow_type(self, arrow_type, expected):
+ assert detect_attribute_type_from_arrow_field(pa.field("f", arrow_type)) is (
+ expected
+ )
+
+ @pytest.mark.parametrize("unit", ["s", "ms", "us", "ns"])
+ def test_timestamp_detected_regardless_of_unit(self, unit):
+ """Detection keys off the Arrow type id, which ignores the time unit."""
+ field = pa.field("f", pa.timestamp(unit))
+ assert detect_attribute_type_from_arrow_field(field) == AttributeType.TIMESTAMP
+
+ def test_large_binary_metadata_overrides_the_arrow_type(self):
+ """LARGE_BINARY travels as a plain string, so only metadata identifies it."""
+ field = pa.field("f", pa.string(), metadata=LARGE_BINARY_METADATA)
+ assert (
+ detect_attribute_type_from_arrow_field(field) == AttributeType.LARGE_BINARY
+ )
+
+ def test_string_without_metadata_is_not_large_binary(self):
+ field = pa.field("f", pa.string())
+ assert detect_attribute_type_from_arrow_field(field) == AttributeType.STRING
+
+ def test_metadata_wins_over_a_non_string_arrow_type(self):
+ """The metadata check runs first and short-circuits the type mapping."""
+ field = pa.field("f", pa.binary(), metadata=LARGE_BINARY_METADATA)
+ assert (
+ detect_attribute_type_from_arrow_field(field) == AttributeType.LARGE_BINARY
+ )
+
+ @pytest.mark.parametrize(
+ "metadata",
+ [
+ pytest.param({}, id="empty"),
+ pytest.param({b"unrelated": b"LARGE_BINARY"}, id="wrong-key"),
+ pytest.param({TEXERA_TYPE_METADATA_KEY: b"STRING"}, id="wrong-value"),
+ pytest.param({TEXERA_TYPE_METADATA_KEY: b""}, id="empty-value"),
+ pytest.param(
+ {TEXERA_TYPE_METADATA_KEY: b"large_binary"}, id="wrong-case-value"
+ ),
+ ],
+ )
+ def test_metadata_that_does_not_mark_large_binary_falls_back_to_type(
+ self, metadata
+ ):
+ field = pa.field("f", pa.string(), metadata=metadata)
+ assert detect_attribute_type_from_arrow_field(field) == AttributeType.STRING
+
+ def test_metadata_accepts_str_keys_since_arrow_encodes_them_as_bytes(self):
+ """pyarrow normalises str metadata to bytes, so a str key still matches."""
+ field = pa.field("f", pa.string(), metadata={"texera_type": "LARGE_BINARY"})
+ assert (
+ detect_attribute_type_from_arrow_field(field) == AttributeType.LARGE_BINARY
+ )
+
+ @pytest.mark.parametrize(
+ "arrow_type",
+ [
+ pytest.param(pa.float32(), id="float32"),
+ pytest.param(pa.int8(), id="int8"),
+ pytest.param(pa.uint64(), id="uint64"),
+ pytest.param(pa.date32(), id="date32"),
+ pytest.param(pa.null(), id="null"),
+ pytest.param(pa.list_(pa.int32()), id="list"),
+ pytest.param(pa.decimal128(10, 2), id="decimal"),
+ ],
+ )
+ def test_unsupported_arrow_type_raises_key_error(self, arrow_type):
+ """Unmapped Arrow types surface as KeyError rather than a wrong type."""
+ field = pa.field("f", arrow_type)
+ with pytest.raises(KeyError):
+ detect_attribute_type_from_arrow_field(field)
+
+ def test_unsupported_arrow_type_still_detected_when_marked_large_binary(self):
+ """The metadata short-circuit runs before the unsupported-type lookup."""
+ field = pa.field("f", pa.float32(), metadata=LARGE_BINARY_METADATA)
+ assert (
+ detect_attribute_type_from_arrow_field(field) == AttributeType.LARGE_BINARY
+ )
+
+
+class TestCreateArrowFieldWithMetadata:
+ @pytest.mark.parametrize(
+ "attr_type, expected_arrow_type",
+ [
+ (AttributeType.INT, pa.int32()),
+ (AttributeType.LONG, pa.int64()),
+ (AttributeType.STRING, pa.string()),
+ (AttributeType.DOUBLE, pa.float64()),
+ (AttributeType.BOOL, pa.bool_()),
+ (AttributeType.BINARY, pa.binary()),
+ (AttributeType.TIMESTAMP, pa.timestamp("us")),
+ (AttributeType.LARGE_BINARY, pa.string()),
+ ],
+ )
+ def test_maps_each_attribute_type_to_its_arrow_type(
+ self, attr_type, expected_arrow_type
+ ):
+ field = create_arrow_field_with_metadata("f", attr_type)
+ assert field.name == "f"
+ assert field.type == expected_arrow_type
+
+ def test_large_binary_field_carries_marker_metadata(self):
+ field = create_arrow_field_with_metadata("payload", AttributeType.LARGE_BINARY)
+ assert field.metadata == LARGE_BINARY_METADATA
+
Review Comment:
`pyarrow.Field.metadata` is a dict-like mapping and may not compare equal to a plain `dict` across PyArrow versions. This assertion is more robust if it checks the marker key/value rather than full mapping equality.
This issue also appears on line 175 of the same file.
##########
amber/src/test/python/core/models/schema/test_attribute_type_utils.py:
##########
@@ -0,0 +1,220 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pyarrow as pa
+import pytest
+
+from core.models.schema.attribute_type import AttributeType
+from core.models.schema.attribute_type_utils import (
+ LARGE_BINARY_METADATA_VALUE,
+ TEXERA_TYPE_METADATA_KEY,
+ create_arrow_field_with_metadata,
+ detect_attribute_type_from_arrow_field,
+)
+
+LARGE_BINARY_METADATA = {TEXERA_TYPE_METADATA_KEY: LARGE_BINARY_METADATA_VALUE}
+
+
+class TestDetectAttributeTypeFromArrowField:
+ @pytest.mark.parametrize(
+ "arrow_type, expected",
+ [
+ (pa.int32(), AttributeType.INT),
+ (pa.int64(), AttributeType.LONG),
+ (pa.string(), AttributeType.STRING),
+ (pa.large_string(), AttributeType.STRING),
+ (pa.float64(), AttributeType.DOUBLE),
+ (pa.bool_(), AttributeType.BOOL),
+ (pa.binary(), AttributeType.BINARY),
+ (pa.large_binary(), AttributeType.BINARY),
+ (pa.timestamp("us"), AttributeType.TIMESTAMP),
+ ],
+ )
+ def test_detects_each_supported_arrow_type(self, arrow_type, expected):
+ assert detect_attribute_type_from_arrow_field(pa.field("f", arrow_type)) is (
+ expected
+ )
+
+ @pytest.mark.parametrize("unit", ["s", "ms", "us", "ns"])
+ def test_timestamp_detected_regardless_of_unit(self, unit):
+ """Detection keys off the Arrow type id, which ignores the time unit."""
+ field = pa.field("f", pa.timestamp(unit))
+ assert detect_attribute_type_from_arrow_field(field) == AttributeType.TIMESTAMP
+
+ def test_large_binary_metadata_overrides_the_arrow_type(self):
+ """LARGE_BINARY travels as a plain string, so only metadata identifies it."""
+ field = pa.field("f", pa.string(), metadata=LARGE_BINARY_METADATA)
+ assert (
+ detect_attribute_type_from_arrow_field(field) == AttributeType.LARGE_BINARY
+ )
+
+ def test_string_without_metadata_is_not_large_binary(self):
+ field = pa.field("f", pa.string())
+ assert detect_attribute_type_from_arrow_field(field) == AttributeType.STRING
+
+ def test_metadata_wins_over_a_non_string_arrow_type(self):
+ """The metadata check runs first and short-circuits the type mapping."""
+ field = pa.field("f", pa.binary(), metadata=LARGE_BINARY_METADATA)
+ assert (
+ detect_attribute_type_from_arrow_field(field) == AttributeType.LARGE_BINARY
+ )
+
+ @pytest.mark.parametrize(
+ "metadata",
+ [
+ pytest.param({}, id="empty"),
+ pytest.param({b"unrelated": b"LARGE_BINARY"}, id="wrong-key"),
+ pytest.param({TEXERA_TYPE_METADATA_KEY: b"STRING"}, id="wrong-value"),
+ pytest.param({TEXERA_TYPE_METADATA_KEY: b""}, id="empty-value"),
+ pytest.param(
+ {TEXERA_TYPE_METADATA_KEY: b"large_binary"}, id="wrong-case-value"
+ ),
+ ],
+ )
+ def test_metadata_that_does_not_mark_large_binary_falls_back_to_type(
+ self, metadata
+ ):
+ field = pa.field("f", pa.string(), metadata=metadata)
+ assert detect_attribute_type_from_arrow_field(field) == AttributeType.STRING
+
+ def test_metadata_accepts_str_keys_since_arrow_encodes_them_as_bytes(self):
+ """pyarrow normalises str metadata to bytes, so a str key still matches."""
+ field = pa.field("f", pa.string(), metadata={"texera_type": "LARGE_BINARY"})
+ assert (
+ detect_attribute_type_from_arrow_field(field) == AttributeType.LARGE_BINARY
+ )
+
+ @pytest.mark.parametrize(
+ "arrow_type",
+ [
+ pytest.param(pa.float32(), id="float32"),
+ pytest.param(pa.int8(), id="int8"),
+ pytest.param(pa.uint64(), id="uint64"),
+ pytest.param(pa.date32(), id="date32"),
+ pytest.param(pa.null(), id="null"),
+ pytest.param(pa.list_(pa.int32()), id="list"),
+ pytest.param(pa.decimal128(10, 2), id="decimal"),
+ ],
+ )
+ def test_unsupported_arrow_type_raises_key_error(self, arrow_type):
+ """Unmapped Arrow types surface as KeyError rather than a wrong type."""
+ field = pa.field("f", arrow_type)
+ with pytest.raises(KeyError):
+ detect_attribute_type_from_arrow_field(field)
+
+ def test_unsupported_arrow_type_still_detected_when_marked_large_binary(self):
+ """The metadata short-circuit runs before the unsupported-type lookup."""
+ field = pa.field("f", pa.float32(), metadata=LARGE_BINARY_METADATA)
+ assert (
+ detect_attribute_type_from_arrow_field(field) == AttributeType.LARGE_BINARY
+ )
+
+
+class TestCreateArrowFieldWithMetadata:
+ @pytest.mark.parametrize(
+ "attr_type, expected_arrow_type",
+ [
+ (AttributeType.INT, pa.int32()),
+ (AttributeType.LONG, pa.int64()),
+ (AttributeType.STRING, pa.string()),
+ (AttributeType.DOUBLE, pa.float64()),
+ (AttributeType.BOOL, pa.bool_()),
+ (AttributeType.BINARY, pa.binary()),
+ (AttributeType.TIMESTAMP, pa.timestamp("us")),
+ (AttributeType.LARGE_BINARY, pa.string()),
+ ],
+ )
+ def test_maps_each_attribute_type_to_its_arrow_type(
+ self, attr_type, expected_arrow_type
+ ):
+ field = create_arrow_field_with_metadata("f", attr_type)
+ assert field.name == "f"
+ assert field.type == expected_arrow_type
+
+ def test_large_binary_field_carries_marker_metadata(self):
+ field = create_arrow_field_with_metadata("payload", AttributeType.LARGE_BINARY)
+ assert field.metadata == LARGE_BINARY_METADATA
+
+ @pytest.mark.parametrize(
+ "attr_type",
+ [t for t in AttributeType if t is not AttributeType.LARGE_BINARY],
+ ids=lambda t: t.name,
+ )
+ def test_non_large_binary_fields_carry_no_metadata(self, attr_type):
+ assert create_arrow_field_with_metadata("f", attr_type).metadata is None
+
+ @pytest.mark.parametrize(
+ "attr_name",
+ [
+ pytest.param("", id="empty"),
+ pytest.param(" ", id="whitespace"),
+ pytest.param("列名", id="unicode"),
+ pytest.param("emoji-🎉", id="emoji"),
+ pytest.param("with space", id="with-space"),
+ pytest.param("a" * 1000, id="very-long"),
+ ],
+ )
+ def test_attribute_name_is_preserved_verbatim(self, attr_name):
+ field = create_arrow_field_with_metadata(attr_name, AttributeType.STRING)
+ assert field.name == attr_name
+
+ def test_unicode_name_survives_on_a_large_binary_field(self):
+ field = create_arrow_field_with_metadata("列名", AttributeType.LARGE_BINARY)
+ assert field.name == "列名"
+ assert field.metadata == LARGE_BINARY_METADATA
+
+ @pytest.mark.parametrize(
+ "invalid_type",
+ [
+ pytest.param("STRING", id="raw-string"),
+ pytest.param(None, id="none"),
+ pytest.param(1, id="int"),
+ pytest.param(pa.string(), id="arrow-type"),
+ ],
+ )
+ def test_non_attribute_type_raises_key_error(self, invalid_type):
+ """Anything that isn't an AttributeType member fails the mapping lookup."""
+ with pytest.raises(KeyError):
+ create_arrow_field_with_metadata("f", invalid_type)
+
Review Comment:
This test over-specifies the exception type for invalid inputs. Depending on whether the provided invalid type is hashable (e.g., some PyArrow types), `TO_ARROW_MAPPING[attr_type]` can raise `TypeError` instead of `KeyError`. If the goal is simply to ensure non-`AttributeType` inputs fail, allow both exception types (or remove the PyArrow-type case).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(pyamber): add unit tests for attribute_type_utils [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #6833: test(pyamber): add unit tests for attribute_type_utils
URL: https://github.com/apache/texera/pull/6833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(pyamber): add unit tests for attribute_type_utils [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6833:
URL: https://github.com/apache/texera/pull/6833#issuecomment-5657812746
This pull request has been automatically closed due to 37 days of inactivity.
If you would like to continue this work, please re-open the PR or open a new
one referencing this. Thank you for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org