You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/08/20 23:52:58 UTC
(spark) branch branch-4.x updated: [SPARK-58906][ML][PYTHON] Make vector and matrix UDTs singletons
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 4937c8ccdd58 [SPARK-58906][ML][PYTHON] Make vector and matrix UDTs singletons
4937c8ccdd58 is described below
commit 4937c8ccdd58d3664caf4cd501ff430d0f278b22
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Fri Aug 21 07:52:28 2026 +0800
[SPARK-58906][ML][PYTHON] Make vector and matrix UDTs singletons
### What changes were proposed in this pull request?
This PR makes the Python ML vector and matrix UDT implementations singletons by using
the existing `DataTypeSingleton` metaclass:
- `pyspark.ml.linalg.VectorUDT`
- `pyspark.ml.linalg.MatrixUDT`
- `pyspark.mllib.linalg.VectorUDT`
- `pyspark.mllib.linalg.MatrixUDT`
It also adds tests that repeated constructor calls and schema JSON round-trips return the
same UDT instance.
This follows the same singleton reuse direction as
https://github.com/apache/spark/pull/57757, which reused the canonical ML vector data
type to avoid repeated equivalent UDT allocations.
### Why are the changes needed?
These UDT classes are stateless. Reusing one Python instance avoids repeated equivalent
allocations and keeps schema round-trips consistent with the singleton behavior already
used by stateless SQL data types.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Added focused tests for both `pyspark.ml.linalg` and `pyspark.mllib.linalg` UDTs.
Also ran:
```
git diff --check
grep -rn -P "[^\x00-\x7F]" python/pyspark/ml/linalg/__init__.py python/pyspark/mllib/linalg/__init__.py python/pyspark/ml/tests/test_linalg.py python/pyspark/mllib/tests/test_linalg.py
awk 'length>100 && $0 !~ /^[[:space:]]*(import|package) / && $0 !~ /https?:\/\// {print FILENAME":"FNR": "length" chars"}' python/pyspark/ml/linalg/__init__.py python/pyspark/mllib/linalg/__init__.py python/pyspark/ml/tests/test_linalg.py python/pyspark/mllib/tests/test_linalg.py
```
The full PySpark test suites were not run locally.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #58166 from zhengruifeng/py-ml-udt-singletons.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
(cherry picked from commit 0cedfe547eae06097529699268b199f7cd4f7d50)
Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
---
python/pyspark/ml/linalg/__init__.py | 5 +++--
python/pyspark/ml/tests/test_linalg.py | 8 ++++++++
python/pyspark/mllib/linalg/__init__.py | 5 +++--
python/pyspark/mllib/tests/test_linalg.py | 8 ++++++++
4 files changed, 22 insertions(+), 4 deletions(-)
diff --git a/python/pyspark/ml/linalg/__init__.py b/python/pyspark/ml/linalg/__init__.py
index a5bf8318c43b..4a07edbadf40 100644
--- a/python/pyspark/ml/linalg/__init__.py
+++ b/python/pyspark/ml/linalg/__init__.py
@@ -45,6 +45,7 @@ from typing import (
import numpy as np
from pyspark.sql.types import (
+ DataTypeSingleton,
UserDefinedType,
StructField,
StructType,
@@ -157,7 +158,7 @@ def _double_to_long_bits(value: float) -> int:
return struct.unpack("Q", struct.pack("d", value))[0]
-class VectorUDT(UserDefinedType):
+class VectorUDT(UserDefinedType, metaclass=DataTypeSingleton):
"""
SQL user-defined type (UDT) for Vector.
"""
@@ -212,7 +213,7 @@ class VectorUDT(UserDefinedType):
return "vector"
-class MatrixUDT(UserDefinedType):
+class MatrixUDT(UserDefinedType, metaclass=DataTypeSingleton):
"""
SQL user-defined type (UDT) for Matrix.
"""
diff --git a/python/pyspark/ml/tests/test_linalg.py b/python/pyspark/ml/tests/test_linalg.py
index 4d5752cecd67..17185ad7b5fd 100644
--- a/python/pyspark/ml/tests/test_linalg.py
+++ b/python/pyspark/ml/tests/test_linalg.py
@@ -328,6 +328,10 @@ class VectorUDTTests(MLlibTestCase):
def test_json_schema(self):
self.assertEqual(VectorUDT.fromJson(self.udt.jsonValue()), self.udt)
+ def test_singleton(self):
+ self.assertIs(VectorUDT(), VectorUDT())
+ self.assertIs(VectorUDT.fromJson(self.udt.jsonValue()), self.udt)
+
def test_serialization(self):
for v in [self.dv0, self.dv1, self.sv0, self.sv1]:
self.assertEqual(v, self.udt.deserialize(self.udt.serialize(v)))
@@ -377,6 +381,10 @@ class MatrixUDTTests(MLlibTestCase):
def test_json_schema(self):
self.assertEqual(MatrixUDT.fromJson(self.udt.jsonValue()), self.udt)
+ def test_singleton(self):
+ self.assertIs(MatrixUDT(), MatrixUDT())
+ self.assertIs(MatrixUDT.fromJson(self.udt.jsonValue()), self.udt)
+
def test_serialization(self):
for m in [self.dm1, self.dm2, self.sm1, self.sm2]:
self.assertEqual(m, self.udt.deserialize(self.udt.serialize(m)))
diff --git a/python/pyspark/mllib/linalg/__init__.py b/python/pyspark/mllib/linalg/__init__.py
index b8b1cbba0fff..363f5c9c8ab7 100644
--- a/python/pyspark/mllib/linalg/__init__.py
+++ b/python/pyspark/mllib/linalg/__init__.py
@@ -49,6 +49,7 @@ import numpy as np
from pyspark import since
from pyspark.ml import linalg as newlinalg
from pyspark.sql.types import (
+ DataTypeSingleton,
UserDefinedType,
StructField,
StructType,
@@ -167,7 +168,7 @@ def _double_to_long_bits(value: float) -> int:
return struct.unpack("Q", struct.pack("d", value))[0]
-class VectorUDT(UserDefinedType):
+class VectorUDT(UserDefinedType, metaclass=DataTypeSingleton):
"""
SQL user-defined type (UDT) for Vector.
"""
@@ -222,7 +223,7 @@ class VectorUDT(UserDefinedType):
return "vector"
-class MatrixUDT(UserDefinedType):
+class MatrixUDT(UserDefinedType, metaclass=DataTypeSingleton):
"""
SQL user-defined type (UDT) for Matrix.
"""
diff --git a/python/pyspark/mllib/tests/test_linalg.py b/python/pyspark/mllib/tests/test_linalg.py
index 36a1e30c0910..00f5f1792cf9 100644
--- a/python/pyspark/mllib/tests/test_linalg.py
+++ b/python/pyspark/mllib/tests/test_linalg.py
@@ -418,6 +418,10 @@ class VectorUDTTests(MLlibTestCase):
def test_json_schema(self):
self.assertEqual(VectorUDT.fromJson(self.udt.jsonValue()), self.udt)
+ def test_singleton(self):
+ self.assertIs(VectorUDT(), VectorUDT())
+ self.assertIs(VectorUDT.fromJson(self.udt.jsonValue()), self.udt)
+
def test_serialization(self):
for v in [self.dv0, self.dv1, self.sv0, self.sv1]:
self.assertEqual(v, self.udt.deserialize(self.udt.serialize(v)))
@@ -479,6 +483,10 @@ class MatrixUDTTests(MLlibTestCase):
def test_json_schema(self):
self.assertEqual(MatrixUDT.fromJson(self.udt.jsonValue()), self.udt)
+ def test_singleton(self):
+ self.assertIs(MatrixUDT(), MatrixUDT())
+ self.assertIs(MatrixUDT.fromJson(self.udt.jsonValue()), self.udt)
+
def test_serialization(self):
for m in [self.dm1, self.dm2, self.sm1, self.sm2]:
self.assertEqual(m, self.udt.deserialize(self.udt.serialize(m)))
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org