You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/10/06 15:31:47 UTC
[GitHub] [iceberg] jun-he opened a new pull request #3234: Support all iceberg types in python library
jun-he opened a new pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234
WIP
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r733331172
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
+ INTEGER = {"java_class": "Integer.class", "python_class": int, "id": 2}
+ LONG = {"java_class": "Long.class", "python_class": int, "id": 3}
+ FLOAT = {"java_class": "Float.class", "python_class": float, "id": 4}
+ DOUBLE = {"java_class": "Double.class", "python_class": float, "id": 5}
+ DATE = {"java_class": "Integer.class", "python_class": int, "id": 6}
+ TIME = {"java_class": "Long.class", "python_class": int, "id": 7}
+ TIMESTAMP = {"java_class": "Long.class", "python_class": int, "id": 8}
+ STRING = {"java_class": "CharSequence.class", "python_class": str, "id": 9}
+ UUID = {"java_class": "java.util.UUID.class", "python_class": uuid.UUID, "id": 10}
+ FIXED = {"java_class": "ByteBuffer.class", "python_class": bytes, "id": 11}
+ BINARY = {"java_class": "ByteBuffer.class", "python_class": bytearray, "id": 12}
+ DECIMAL = {"java_class": "BigDecimal.class", "python_class": Decimal, "id": 13}
+ STRUCT = {"java_class": "Void.class", "python_class": None, "id": 14}
+ LIST = {"java_class": "Void.class", "python_class": None, "id": 15}
+ MAP = {"java_class": "Void.class", "python_class": None, "id": 16}
+
+
+class Type(object):
+ length: int
+ scale: int
+ precision: int
+
+ def __init__(self):
+ pass
+
+ def type_id(self):
+ pass
+
+ def is_primitive_type(self):
+ return False
+
+ def as_primitive_type(self):
+ raise ValueError("Not a primitive type: " + self)
+
+ def as_struct_type(self):
+ raise ValueError("Not a struct type: " + self)
+
+ def as_list_type(self):
+ raise ValueError("Not a list type: " + self)
+
+ def asMapType(self):
+ raise ValueError("Not a map type: " + self)
+
+ def is_nested_type(self):
+ return False
+
+ def is_struct_type(self):
+ return False
+
+ def is_list_type(self):
+ return False
+
+ def is_map_type(self):
+ return False
+
+ def as_nested_type(self):
+ raise ValueError("Not a nested type: " + self)
+
+
+class PrimitiveType(Type):
+
+ def __eq__(self, other):
+ return type(self) == type(other)
+
+ def __ne__(self, other):
+ return not self.__eq__(other)
Review comment:
What I meant is that the default one in python3 is slightly different from this one in the python legacy module.
This implementation only works if `type(self)` implements `__eq__`.
The default one in python 3 is better and can handle other cases. IIUC, it works like
```
def __ne__(self, other):
result = self.__eq__(other)
if result is NotImplemented:
return NotImplemented
return not result
```
So we should remove `__ne__` here if we want to support `!=`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
jun-he commented on pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#issuecomment-948272261
@rdblue @pdames @samredai I updated the PR and added tests. Can you take another look? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r734090591
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self) -> bool:
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType({length})", is_primitive=True)
+ self._length = length
+
+ @property
+ def length(self) -> int:
+ return self._length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ @property
+ def precision(self) -> int:
+ return self._precision
+
+ @property
+ def scale(self) -> int:
+ return self._scale
+
+
+class NestedField(object):
+ def __init__(self, is_optional: bool, field_id: int, name: str, field_type: Type, doc=None):
+ self._is_optional = is_optional
+ self._id = field_id
+ self._name = name
+ self._type = field_type
+ self._doc = doc
+
+ @property
+ def is_optional(self) -> bool:
+ return self._is_optional
+
+ @property
+ def is_required(self) -> bool:
+ return not self._is_optional
+
+ @property
+ def field_id(self) -> int:
+ return self._id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ @property
+ def type(self) -> Type:
+ return self._type
+
+ def __repr__(self):
+ return (f"NestedField({self._is_optional}, {self._id}, "
+ f"{repr(self._name)}, {repr(self._type)}, {repr(self._doc)})")
+
+ def __str__(self):
+ return (f"{self._id}: {self._name}: {'optional' if self._is_optional else 'required'} {self._type}"
+ "" if self._doc is None else f" ({self._doc})")
+
+
+class StructType(Type):
+ def __init__(self, fields: list):
+ super().__init__(f"struct<{', '.join(map(str, fields))}>", f"StructType({repr(fields)})")
+ self._fields = fields
+
+ @property
+ def fields(self) -> list:
+ return self._fields
+
+
+class ListType(Type):
+ def __init__(self, element_field: NestedField):
+ super().__init__(f"list<{element_field.type}>", f"ListType({repr(element_field)})")
+ self._element_field = element_field
+
+ @property
+ def element(self) -> NestedField:
+ return self._element_field
+
+
+class MapType(Type):
+ def __init__(self, key_field: NestedField, value_field: NestedField):
+ super().__init__(f"map<{key_field.type}, {value_field.type}>",
+ f"MapType({repr(key_field)}, {repr(value_field)})")
+ self._key_field = key_field
Review comment:
Yeah, I think key, value, and element are probably better names for the constructor args.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r723862517
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
+ INTEGER = {"java_class": "Integer.class", "python_class": int, "id": 2}
+ LONG = {"java_class": "Long.class", "python_class": int, "id": 3}
+ FLOAT = {"java_class": "Float.class", "python_class": float, "id": 4}
+ DOUBLE = {"java_class": "Double.class", "python_class": float, "id": 5}
+ DATE = {"java_class": "Integer.class", "python_class": int, "id": 6}
+ TIME = {"java_class": "Long.class", "python_class": int, "id": 7}
+ TIMESTAMP = {"java_class": "Long.class", "python_class": int, "id": 8}
+ STRING = {"java_class": "CharSequence.class", "python_class": str, "id": 9}
+ UUID = {"java_class": "java.util.UUID.class", "python_class": uuid.UUID, "id": 10}
+ FIXED = {"java_class": "ByteBuffer.class", "python_class": bytes, "id": 11}
+ BINARY = {"java_class": "ByteBuffer.class", "python_class": bytearray, "id": 12}
+ DECIMAL = {"java_class": "BigDecimal.class", "python_class": Decimal, "id": 13}
+ STRUCT = {"java_class": "Void.class", "python_class": None, "id": 14}
+ LIST = {"java_class": "Void.class", "python_class": None, "id": 15}
+ MAP = {"java_class": "Void.class", "python_class": None, "id": 16}
+
+
+class Type(object):
+ length: int
+ scale: int
+ precision: int
+
+ def __init__(self):
+ pass
+
+ def type_id(self):
+ pass
+
+ def is_primitive_type(self):
+ return False
+
+ def as_primitive_type(self):
+ raise ValueError("Not a primitive type: " + self)
+
+ def as_struct_type(self):
+ raise ValueError("Not a struct type: " + self)
+
+ def as_list_type(self):
+ raise ValueError("Not a list type: " + self)
+
+ def asMapType(self):
Review comment:
Yep, that is from the legacy library and will clean up those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r733265984
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self):
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType[{length}]", is_primitive=True)
Review comment:
Thanks for the comment and updated accordingly.
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self):
+ return self._is_primitive
+
+
+class FixedType(Type):
Review comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
samredai commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r734070120
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self):
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType[{length}]", is_primitive=True)
+ self._length = length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ def precision(self):
+ return self._precision
+
+ def scale(self):
+ return self._scale
+
+
+class NestedField(object):
+ def __init__(self, is_optional: bool, field_id: int, name: str, field_type: Type, doc=None):
+ self._is_optional = is_optional
+ self._id = field_id
+ self._name = name
+ self._type = field_type
+ self._doc = doc
+
+ @property
+ def is_optional(self):
+ return self._is_optional
+
+ @property
+ def is_required(self):
+ return not self._is_optional
+
+ @property
+ def field_id(self):
+ return self._id
+
+ @property
+ def type(self):
+ return self._type
+
+ def __repr__(self):
Review comment:
what do you think of having the named arguments in the repr?
```py
return (f"NestedField(is_optional={self._is_optional}, field_id={self._id}, "
f"name={repr(self._name)}, field_type={repr(self._type)}, doc={repr(self._doc)})")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
samredai commented on pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#issuecomment-948007271
@jun-he do you think you could give me access to this branch `jun-he:jun/add-types` in your forked repo? I have a commit queued up to add test coverage to pass CI and figured we can do that in the same PR. I also added some of the additional linters/autoformatting we discussed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r723415943
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
Review comment:
Do we need the Java class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r734266696
##########
File path: python/tests/test_types.py
##########
@@ -0,0 +1,85 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from iceberg.types import (BinaryType, BooleanType, DateType, DecimalType, DoubleType, FixedType,
+ FloatType, IntegerType, ListType, LongType, MapType, NestedField, StringType,
+ StructType, TimestampType, TimestamptzType, TimeType, UUIDType)
+import pytest
+
+
+@pytest.mark.parametrize("input_type",
+ [BooleanType, IntegerType, LongType, FloatType, DoubleType, DateType, TimeType,
+ TimestampType, TimestamptzType, StringType, UUIDType, BinaryType])
Review comment:
Those are tested in individual tests later with additional asserts in addition to eval of repr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#issuecomment-949050782
@samredai, if we can format the code here, let's do that. But I wouldn't say let's update the config for formatting, since that's a separate change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r734089383
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self) -> bool:
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType({length})", is_primitive=True)
+ self._length = length
+
+ @property
+ def length(self) -> int:
+ return self._length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ @property
+ def precision(self) -> int:
+ return self._precision
+
+ @property
+ def scale(self) -> int:
+ return self._scale
+
+
+class NestedField(object):
+ def __init__(self, is_optional: bool, field_id: int, name: str, field_type: Type, doc=None):
Review comment:
Should doc have a type annotation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r723417490
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
+ INTEGER = {"java_class": "Integer.class", "python_class": int, "id": 2}
+ LONG = {"java_class": "Long.class", "python_class": int, "id": 3}
+ FLOAT = {"java_class": "Float.class", "python_class": float, "id": 4}
+ DOUBLE = {"java_class": "Double.class", "python_class": float, "id": 5}
+ DATE = {"java_class": "Integer.class", "python_class": int, "id": 6}
+ TIME = {"java_class": "Long.class", "python_class": int, "id": 7}
+ TIMESTAMP = {"java_class": "Long.class", "python_class": int, "id": 8}
+ STRING = {"java_class": "CharSequence.class", "python_class": str, "id": 9}
+ UUID = {"java_class": "java.util.UUID.class", "python_class": uuid.UUID, "id": 10}
+ FIXED = {"java_class": "ByteBuffer.class", "python_class": bytes, "id": 11}
+ BINARY = {"java_class": "ByteBuffer.class", "python_class": bytearray, "id": 12}
+ DECIMAL = {"java_class": "BigDecimal.class", "python_class": Decimal, "id": 13}
+ STRUCT = {"java_class": "Void.class", "python_class": None, "id": 14}
+ LIST = {"java_class": "Void.class", "python_class": None, "id": 15}
+ MAP = {"java_class": "Void.class", "python_class": None, "id": 16}
+
+
+class Type(object):
+ length: int
+ scale: int
+ precision: int
Review comment:
Do these need to be here or on the specific types, fixed and decimal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r734088464
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self):
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType[{length}]", is_primitive=True)
+ self._length = length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ def precision(self):
+ return self._precision
+
+ def scale(self):
+ return self._scale
+
+
+class NestedField(object):
+ def __init__(self, is_optional: bool, field_id: int, name: str, field_type: Type, doc=None):
+ self._is_optional = is_optional
+ self._id = field_id
+ self._name = name
+ self._type = field_type
+ self._doc = doc
+
+ @property
+ def is_optional(self):
+ return self._is_optional
+
+ @property
+ def is_required(self):
+ return not self._is_optional
+
+ @property
+ def field_id(self):
+ return self._id
+
+ @property
+ def type(self):
+ return self._type
+
+ def __repr__(self):
Review comment:
I'm all for it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
jun-he commented on pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#issuecomment-949327568
@samredai that's a great idea to have the formatting. We have an issue (https://github.com/apache/iceberg/issues/3282) for that. I agree that we should have it ASAP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r733266179
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self):
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType[{length}]", is_primitive=True)
+ self._length = length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ def precision(self):
+ return self._precision
+
+ def scale(self):
+ return self._scale
+
+
+class NestedField(object):
+ def __init__(self, is_optional: bool, field_id: int, name: str, field_type: Type, doc=None):
+ self._is_optional = is_optional
+ self._id = field_id
+ self._name = name
+ self._type = field_type
+ self._doc = doc
+
+ @property
+ def is_optional(self):
+ return self._is_optional
+
+ @property
+ def is_required(self):
+ return not self._is_optional
+
+ @property
+ def field_id(self):
+ return self._id
+
+ @property
+ def type(self):
+ return self._type
+
+ def __repr__(self):
Review comment:
Yep, updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#issuecomment-949053902
Nice work, @jun-he! There are only nits left, so I'm going to commit this to keep it moving. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r723862324
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
Review comment:
That is used in Java library for `partition_spec`'s `lazyJavaClasses` or object cast in Accessors.
Seems no need here as python will not cast data to a Java object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
samredai commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r729969776
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
+ INTEGER = {"java_class": "Integer.class", "python_class": int, "id": 2}
+ LONG = {"java_class": "Long.class", "python_class": int, "id": 3}
+ FLOAT = {"java_class": "Float.class", "python_class": float, "id": 4}
+ DOUBLE = {"java_class": "Double.class", "python_class": float, "id": 5}
+ DATE = {"java_class": "Integer.class", "python_class": int, "id": 6}
+ TIME = {"java_class": "Long.class", "python_class": int, "id": 7}
+ TIMESTAMP = {"java_class": "Long.class", "python_class": int, "id": 8}
+ STRING = {"java_class": "CharSequence.class", "python_class": str, "id": 9}
+ UUID = {"java_class": "java.util.UUID.class", "python_class": uuid.UUID, "id": 10}
+ FIXED = {"java_class": "ByteBuffer.class", "python_class": bytes, "id": 11}
+ BINARY = {"java_class": "ByteBuffer.class", "python_class": bytearray, "id": 12}
+ DECIMAL = {"java_class": "BigDecimal.class", "python_class": Decimal, "id": 13}
+ STRUCT = {"java_class": "Void.class", "python_class": None, "id": 14}
+ LIST = {"java_class": "Void.class", "python_class": None, "id": 15}
+ MAP = {"java_class": "Void.class", "python_class": None, "id": 16}
+
+
+class Type(object):
+ length: int
+ scale: int
+ precision: int
+
+ def __init__(self):
+ pass
+
+ def type_id(self):
+ pass
+
+ def is_primitive_type(self):
+ return False
+
+ def as_primitive_type(self):
+ raise ValueError("Not a primitive type: " + self)
+
+ def as_struct_type(self):
+ raise ValueError("Not a struct type: " + self)
+
+ def as_list_type(self):
+ raise ValueError("Not a list type: " + self)
+
+ def asMapType(self):
+ raise ValueError("Not a map type: " + self)
+
+ def is_nested_type(self):
+ return False
+
+ def is_struct_type(self):
+ return False
+
+ def is_list_type(self):
+ return False
+
+ def is_map_type(self):
+ return False
+
+ def as_nested_type(self):
+ raise ValueError("Not a nested type: " + self)
+
+
+class PrimitiveType(Type):
+
+ def __eq__(self, other):
+ return type(self) == type(other)
+
+ def __ne__(self, other):
+ return not self.__eq__(other)
Review comment:
+1 I think it's the same. "returns `NotImplemented` when false" is describing the `__eq__` behavior but once you implement that you get `__ne__` for free.
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self):
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType[{length}]", is_primitive=True)
+ self._length = length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ def precision(self):
+ return self._precision
+
+ def scale(self):
+ return self._scale
+
+
+class NestedField(object):
+ def __init__(self, is_optional: bool, field_id: int, name: str, field_type: Type, doc=None):
+ self._is_optional = is_optional
+ self._id = field_id
+ self._name = name
+ self._type = field_type
+ self._doc = doc
+
+ @property
+ def is_optional(self):
+ return self._is_optional
+
+ @property
+ def is_required(self):
+ return not self._is_optional
+
+ @property
+ def field_id(self):
+ return self._id
+
+ @property
+ def type(self):
+ return self._type
+
+ def __repr__(self):
+ return f"{self._id}: {self._name}: {'optional' if self._is_optional else 'required'} {self._type}" \
+ "" if self._doc is None else f" ({self._doc})"
+
+ def __str__(self):
+ return self.__repr__()
+
+
+class StructType(Type):
+ def __init__(self, fields: list):
+ super().__init__(f"struct<{', '.join(map(str, fields))}>", f"StructType<{', '.join(map(str, fields))}>")
+ self._fields = fields
Review comment:
To keep it pythonic, should we avoid accessor methods? Nothing is truly private so if we expect it to be accessed outside of the class maybe we just remove the underscore here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
samredai commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r734059772
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self) -> bool:
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType({length})", is_primitive=True)
+ self._length = length
+
+ @property
+ def length(self) -> int:
+ return self._length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ @property
+ def precision(self) -> int:
+ return self._precision
+
+ @property
+ def scale(self) -> int:
+ return self._scale
+
+
+class NestedField(object):
+ def __init__(self, is_optional: bool, field_id: int, name: str, field_type: Type, doc=None):
+ self._is_optional = is_optional
+ self._id = field_id
+ self._name = name
+ self._type = field_type
+ self._doc = doc
+
+ @property
+ def is_optional(self) -> bool:
+ return self._is_optional
+
+ @property
+ def is_required(self) -> bool:
+ return not self._is_optional
+
+ @property
+ def field_id(self) -> int:
+ return self._id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ @property
+ def type(self) -> Type:
+ return self._type
+
+ def __repr__(self):
+ return (f"NestedField({self._is_optional}, {self._id}, "
+ f"{repr(self._name)}, {repr(self._type)}, {repr(self._doc)})")
+
+ def __str__(self):
+ return (f"{self._id}: {self._name}: {'optional' if self._is_optional else 'required'} {self._type}"
+ "" if self._doc is None else f" ({self._doc})")
+
+
+class StructType(Type):
+ def __init__(self, fields: list):
+ super().__init__(f"struct<{', '.join(map(str, fields))}>", f"StructType({repr(fields)})")
+ self._fields = fields
+
+ @property
+ def fields(self) -> list:
+ return self._fields
+
+
+class ListType(Type):
+ def __init__(self, element_field: NestedField):
+ super().__init__(f"list<{element_field.type}>", f"ListType({repr(element_field)})")
+ self._element_field = element_field
+
+ @property
+ def element(self) -> NestedField:
+ return self._element_field
+
+
+class MapType(Type):
+ def __init__(self, key_field: NestedField, value_field: NestedField):
+ super().__init__(f"map<{key_field.type}, {value_field.type}>",
+ f"MapType({repr(key_field)}, {repr(value_field)})")
+ self._key_field = key_field
Review comment:
small nit: Can this just argument just be `key` and then this line can be `self._key = key`? Same for `value_field` and `element_field`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r734091176
##########
File path: python/tests/test_types.py
##########
@@ -0,0 +1,85 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from iceberg.types import (BinaryType, BooleanType, DateType, DecimalType, DoubleType, FixedType,
+ FloatType, IntegerType, ListType, LongType, MapType, NestedField, StringType,
+ StructType, TimestampType, TimestamptzType, TimeType, UUIDType)
+import pytest
+
+
+@pytest.mark.parametrize("input_type",
+ [BooleanType, IntegerType, LongType, FloatType, DoubleType, DateType, TimeType,
+ TimestampType, TimestamptzType, StringType, UUIDType, BinaryType])
Review comment:
Could we add struct, map, and list cases, too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r733263904
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self):
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType[{length}]", is_primitive=True)
+ self._length = length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ def precision(self):
Review comment:
yep, SG.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r723862324
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
Review comment:
That is used in Java library for `partition_spec`'s `lazyJavaClasses` or object cast in Accessors.
Seems no need here as python will not cast data to a Java object.
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
+ INTEGER = {"java_class": "Integer.class", "python_class": int, "id": 2}
+ LONG = {"java_class": "Long.class", "python_class": int, "id": 3}
+ FLOAT = {"java_class": "Float.class", "python_class": float, "id": 4}
+ DOUBLE = {"java_class": "Double.class", "python_class": float, "id": 5}
+ DATE = {"java_class": "Integer.class", "python_class": int, "id": 6}
+ TIME = {"java_class": "Long.class", "python_class": int, "id": 7}
+ TIMESTAMP = {"java_class": "Long.class", "python_class": int, "id": 8}
+ STRING = {"java_class": "CharSequence.class", "python_class": str, "id": 9}
+ UUID = {"java_class": "java.util.UUID.class", "python_class": uuid.UUID, "id": 10}
+ FIXED = {"java_class": "ByteBuffer.class", "python_class": bytes, "id": 11}
+ BINARY = {"java_class": "ByteBuffer.class", "python_class": bytearray, "id": 12}
+ DECIMAL = {"java_class": "BigDecimal.class", "python_class": Decimal, "id": 13}
+ STRUCT = {"java_class": "Void.class", "python_class": None, "id": 14}
+ LIST = {"java_class": "Void.class", "python_class": None, "id": 15}
+ MAP = {"java_class": "Void.class", "python_class": None, "id": 16}
+
+
+class Type(object):
+ length: int
+ scale: int
+ precision: int
+
+ def __init__(self):
+ pass
+
+ def type_id(self):
+ pass
+
+ def is_primitive_type(self):
+ return False
+
+ def as_primitive_type(self):
+ raise ValueError("Not a primitive type: " + self)
+
+ def as_struct_type(self):
+ raise ValueError("Not a struct type: " + self)
+
+ def as_list_type(self):
+ raise ValueError("Not a list type: " + self)
+
+ def asMapType(self):
Review comment:
Yep, that is from the legacy library and will clean up those.
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
+ INTEGER = {"java_class": "Integer.class", "python_class": int, "id": 2}
+ LONG = {"java_class": "Long.class", "python_class": int, "id": 3}
+ FLOAT = {"java_class": "Float.class", "python_class": float, "id": 4}
+ DOUBLE = {"java_class": "Double.class", "python_class": float, "id": 5}
+ DATE = {"java_class": "Integer.class", "python_class": int, "id": 6}
+ TIME = {"java_class": "Long.class", "python_class": int, "id": 7}
+ TIMESTAMP = {"java_class": "Long.class", "python_class": int, "id": 8}
+ STRING = {"java_class": "CharSequence.class", "python_class": str, "id": 9}
+ UUID = {"java_class": "java.util.UUID.class", "python_class": uuid.UUID, "id": 10}
+ FIXED = {"java_class": "ByteBuffer.class", "python_class": bytes, "id": 11}
+ BINARY = {"java_class": "ByteBuffer.class", "python_class": bytearray, "id": 12}
+ DECIMAL = {"java_class": "BigDecimal.class", "python_class": Decimal, "id": 13}
+ STRUCT = {"java_class": "Void.class", "python_class": None, "id": 14}
+ LIST = {"java_class": "Void.class", "python_class": None, "id": 15}
+ MAP = {"java_class": "Void.class", "python_class": None, "id": 16}
+
+
+class Type(object):
+ length: int
+ scale: int
+ precision: int
Review comment:
Agree and they should not be there for all types.
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
+ INTEGER = {"java_class": "Integer.class", "python_class": int, "id": 2}
+ LONG = {"java_class": "Long.class", "python_class": int, "id": 3}
+ FLOAT = {"java_class": "Float.class", "python_class": float, "id": 4}
+ DOUBLE = {"java_class": "Double.class", "python_class": float, "id": 5}
+ DATE = {"java_class": "Integer.class", "python_class": int, "id": 6}
+ TIME = {"java_class": "Long.class", "python_class": int, "id": 7}
+ TIMESTAMP = {"java_class": "Long.class", "python_class": int, "id": 8}
+ STRING = {"java_class": "CharSequence.class", "python_class": str, "id": 9}
+ UUID = {"java_class": "java.util.UUID.class", "python_class": uuid.UUID, "id": 10}
+ FIXED = {"java_class": "ByteBuffer.class", "python_class": bytes, "id": 11}
+ BINARY = {"java_class": "ByteBuffer.class", "python_class": bytearray, "id": 12}
+ DECIMAL = {"java_class": "BigDecimal.class", "python_class": Decimal, "id": 13}
+ STRUCT = {"java_class": "Void.class", "python_class": None, "id": 14}
+ LIST = {"java_class": "Void.class", "python_class": None, "id": 15}
+ MAP = {"java_class": "Void.class", "python_class": None, "id": 16}
+
+
+class Type(object):
+ length: int
+ scale: int
+ precision: int
+
+ def __init__(self):
+ pass
+
+ def type_id(self):
+ pass
+
+ def is_primitive_type(self):
+ return False
+
+ def as_primitive_type(self):
+ raise ValueError("Not a primitive type: " + self)
+
+ def as_struct_type(self):
+ raise ValueError("Not a struct type: " + self)
+
+ def as_list_type(self):
+ raise ValueError("Not a list type: " + self)
+
+ def asMapType(self):
+ raise ValueError("Not a map type: " + self)
+
+ def is_nested_type(self):
+ return False
+
+ def is_struct_type(self):
+ return False
+
+ def is_list_type(self):
+ return False
+
+ def is_map_type(self):
+ return False
+
+ def as_nested_type(self):
+ raise ValueError("Not a nested type: " + self)
+
+
+class PrimitiveType(Type):
+
+ def __eq__(self, other):
+ return type(self) == type(other)
+
+ def __ne__(self, other):
+ return not self.__eq__(other)
Review comment:
Seems not completely same.
Based on https://docs.python.org/3/reference/datamodel.html#object.__ne__
```
By default, object implements __eq__() by using is, returning NotImplemented in the case of a false comparison: True if x is y else NotImplemented. For __ne__(), by default it delegates to __eq__() and inverts the result unless it is NotImplemented.
```
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
+ INTEGER = {"java_class": "Integer.class", "python_class": int, "id": 2}
+ LONG = {"java_class": "Long.class", "python_class": int, "id": 3}
+ FLOAT = {"java_class": "Float.class", "python_class": float, "id": 4}
+ DOUBLE = {"java_class": "Double.class", "python_class": float, "id": 5}
+ DATE = {"java_class": "Integer.class", "python_class": int, "id": 6}
+ TIME = {"java_class": "Long.class", "python_class": int, "id": 7}
+ TIMESTAMP = {"java_class": "Long.class", "python_class": int, "id": 8}
+ STRING = {"java_class": "CharSequence.class", "python_class": str, "id": 9}
+ UUID = {"java_class": "java.util.UUID.class", "python_class": uuid.UUID, "id": 10}
+ FIXED = {"java_class": "ByteBuffer.class", "python_class": bytes, "id": 11}
+ BINARY = {"java_class": "ByteBuffer.class", "python_class": bytearray, "id": 12}
+ DECIMAL = {"java_class": "BigDecimal.class", "python_class": Decimal, "id": 13}
+ STRUCT = {"java_class": "Void.class", "python_class": None, "id": 14}
+ LIST = {"java_class": "Void.class", "python_class": None, "id": 15}
+ MAP = {"java_class": "Void.class", "python_class": None, "id": 16}
+
+
+class Type(object):
+ length: int
+ scale: int
+ precision: int
+
+ def __init__(self):
+ pass
+
+ def type_id(self):
+ pass
+
+ def is_primitive_type(self):
+ return False
+
+ def as_primitive_type(self):
+ raise ValueError("Not a primitive type: " + self)
+
+ def as_struct_type(self):
+ raise ValueError("Not a struct type: " + self)
+
+ def as_list_type(self):
+ raise ValueError("Not a list type: " + self)
+
+ def asMapType(self):
+ raise ValueError("Not a map type: " + self)
+
+ def is_nested_type(self):
+ return False
+
+ def is_struct_type(self):
+ return False
+
+ def is_list_type(self):
+ return False
+
+ def is_map_type(self):
+ return False
+
+ def as_nested_type(self):
+ raise ValueError("Not a nested type: " + self)
+
+
+class PrimitiveType(Type):
+
+ def __eq__(self, other):
+ return type(self) == type(other)
+
+ def __ne__(self, other):
+ return not self.__eq__(other)
Review comment:
Seems not completely same.
Based on https://docs.python.org/3/reference/datamodel.html#object.__ne__
```
By default, object implements __eq__() by using is, returning NotImplemented in the case of a false comparison: True if x is y else NotImplemented. For __ne__(), by default it delegates to __eq__() and inverts the result unless it is NotImplemented.
```
default will throw `NotImplemented` if false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r723427348
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
+ INTEGER = {"java_class": "Integer.class", "python_class": int, "id": 2}
+ LONG = {"java_class": "Long.class", "python_class": int, "id": 3}
+ FLOAT = {"java_class": "Float.class", "python_class": float, "id": 4}
+ DOUBLE = {"java_class": "Double.class", "python_class": float, "id": 5}
+ DATE = {"java_class": "Integer.class", "python_class": int, "id": 6}
+ TIME = {"java_class": "Long.class", "python_class": int, "id": 7}
+ TIMESTAMP = {"java_class": "Long.class", "python_class": int, "id": 8}
+ STRING = {"java_class": "CharSequence.class", "python_class": str, "id": 9}
+ UUID = {"java_class": "java.util.UUID.class", "python_class": uuid.UUID, "id": 10}
+ FIXED = {"java_class": "ByteBuffer.class", "python_class": bytes, "id": 11}
+ BINARY = {"java_class": "ByteBuffer.class", "python_class": bytearray, "id": 12}
+ DECIMAL = {"java_class": "BigDecimal.class", "python_class": Decimal, "id": 13}
+ STRUCT = {"java_class": "Void.class", "python_class": None, "id": 14}
+ LIST = {"java_class": "Void.class", "python_class": None, "id": 15}
+ MAP = {"java_class": "Void.class", "python_class": None, "id": 16}
+
+
+class Type(object):
+ length: int
+ scale: int
+ precision: int
+
+ def __init__(self):
+ pass
+
+ def type_id(self):
+ pass
+
+ def is_primitive_type(self):
+ return False
+
+ def as_primitive_type(self):
+ raise ValueError("Not a primitive type: " + self)
+
+ def as_struct_type(self):
+ raise ValueError("Not a struct type: " + self)
+
+ def as_list_type(self):
+ raise ValueError("Not a list type: " + self)
+
+ def asMapType(self):
+ raise ValueError("Not a map type: " + self)
+
+ def is_nested_type(self):
+ return False
+
+ def is_struct_type(self):
+ return False
+
+ def is_list_type(self):
+ return False
+
+ def is_map_type(self):
+ return False
+
+ def as_nested_type(self):
+ raise ValueError("Not a nested type: " + self)
+
+
+class PrimitiveType(Type):
+
+ def __eq__(self, other):
+ return type(self) == type(other)
+
+ def __ne__(self, other):
+ return not self.__eq__(other)
+
+ def is_primitive_type(self):
+ return True
+
+ def as_primitive_type(self):
+ return self
+
+
+class NestedType(Type):
+
+ def __init__(self):
+ super(NestedType, self).__init__()
+
+ def is_nested_type(self):
+ return True
+
+ def as_nested_type(self):
+ return self
+
+ def fields(self):
+ pass
+
+ def field_type(self, name):
+ pass
+
+ def field(self, id):
+ pass
+
+
+class BooleanType(PrimitiveType):
+ __instance = None
+
+ @staticmethod
+ def get():
+ if BooleanType.__instance is None:
+ BooleanType()
Review comment:
Should this be `BooleanType.__instance = BooleanType()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
samredai commented on pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#issuecomment-949020334
Should we add the auto formatting stuff in this PR to prevent having to do a huge auto format later? (It's also fine to do in a follow up PR). I was thinking something like:
```ini
[testenv:format]
description = reformat all source code
basepython = python3
deps =
black
isort
flake8
skip_install = true
commands =
isort --recursive --project iceberg --profile black setup.py src tests
black setup.py src tests
flake8 setup.py src tests
[testenv:linters]
basepython = python3
skip_install = true
deps =
.
{[testenv:isort]deps}
{[testenv:black]deps}
{[testenv:flake8]deps}
{[testenv:bandit]deps}
{[testenv:mypy]deps}
commands =
{[testenv:isort]deps}
{[testenv:black]deps}
{[testenv:flake8]commands}
{[testenv:bandit]commands}
{[testenv:mypy]commands}
[testenv:isort]
basepython = python3
skip_install = true
deps =
isort
commands =
isort --recursive --project iceberg --profile black --check-only setup.py src tests
[testenv:black]
basepython = python3
skip_install = true
deps =
black
commands =
black --check --diff src setup.py tests
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r734280896
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self):
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType[{length}]", is_primitive=True)
+ self._length = length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ def precision(self):
+ return self._precision
+
+ def scale(self):
+ return self._scale
+
+
+class NestedField(object):
+ def __init__(self, is_optional: bool, field_id: int, name: str, field_type: Type, doc=None):
+ self._is_optional = is_optional
+ self._id = field_id
+ self._name = name
+ self._type = field_type
+ self._doc = doc
+
+ @property
+ def is_optional(self):
+ return self._is_optional
+
+ @property
+ def is_required(self):
+ return not self._is_optional
+
+ @property
+ def field_id(self):
+ return self._id
+
+ @property
+ def type(self):
+ return self._type
+
+ def __repr__(self):
Review comment:
SG, adding them in https://github.com/apache/iceberg/pull/3350
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self) -> bool:
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType({length})", is_primitive=True)
+ self._length = length
+
+ @property
+ def length(self) -> int:
+ return self._length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ @property
+ def precision(self) -> int:
+ return self._precision
+
+ @property
+ def scale(self) -> int:
+ return self._scale
+
+
+class NestedField(object):
+ def __init__(self, is_optional: bool, field_id: int, name: str, field_type: Type, doc=None):
+ self._is_optional = is_optional
+ self._id = field_id
+ self._name = name
+ self._type = field_type
+ self._doc = doc
+
+ @property
+ def is_optional(self) -> bool:
+ return self._is_optional
+
+ @property
+ def is_required(self) -> bool:
+ return not self._is_optional
+
+ @property
+ def field_id(self) -> int:
+ return self._id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ @property
+ def type(self) -> Type:
+ return self._type
+
+ def __repr__(self):
+ return (f"NestedField({self._is_optional}, {self._id}, "
+ f"{repr(self._name)}, {repr(self._type)}, {repr(self._doc)})")
+
+ def __str__(self):
+ return (f"{self._id}: {self._name}: {'optional' if self._is_optional else 'required'} {self._type}"
+ "" if self._doc is None else f" ({self._doc})")
+
+
+class StructType(Type):
+ def __init__(self, fields: list):
+ super().__init__(f"struct<{', '.join(map(str, fields))}>", f"StructType({repr(fields)})")
+ self._fields = fields
+
+ @property
+ def fields(self) -> list:
+ return self._fields
+
+
+class ListType(Type):
+ def __init__(self, element_field: NestedField):
+ super().__init__(f"list<{element_field.type}>", f"ListType({repr(element_field)})")
+ self._element_field = element_field
+
+ @property
+ def element(self) -> NestedField:
+ return self._element_field
+
+
+class MapType(Type):
+ def __init__(self, key_field: NestedField, value_field: NestedField):
+ super().__init__(f"map<{key_field.type}, {value_field.type}>",
+ f"MapType({repr(key_field)}, {repr(value_field)})")
+ self._key_field = key_field
Review comment:
SG, updating the argument names in https://github.com/apache/iceberg/pull/3350
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self) -> bool:
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType({length})", is_primitive=True)
+ self._length = length
+
+ @property
+ def length(self) -> int:
+ return self._length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ @property
+ def precision(self) -> int:
+ return self._precision
+
+ @property
+ def scale(self) -> int:
+ return self._scale
+
+
+class NestedField(object):
+ def __init__(self, is_optional: bool, field_id: int, name: str, field_type: Type, doc=None):
Review comment:
updated it with str annotation in https://github.com/apache/iceberg/pull/3350
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r723416726
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
+ INTEGER = {"java_class": "Integer.class", "python_class": int, "id": 2}
+ LONG = {"java_class": "Long.class", "python_class": int, "id": 3}
+ FLOAT = {"java_class": "Float.class", "python_class": float, "id": 4}
+ DOUBLE = {"java_class": "Double.class", "python_class": float, "id": 5}
+ DATE = {"java_class": "Integer.class", "python_class": int, "id": 6}
+ TIME = {"java_class": "Long.class", "python_class": int, "id": 7}
+ TIMESTAMP = {"java_class": "Long.class", "python_class": int, "id": 8}
+ STRING = {"java_class": "CharSequence.class", "python_class": str, "id": 9}
+ UUID = {"java_class": "java.util.UUID.class", "python_class": uuid.UUID, "id": 10}
+ FIXED = {"java_class": "ByteBuffer.class", "python_class": bytes, "id": 11}
+ BINARY = {"java_class": "ByteBuffer.class", "python_class": bytearray, "id": 12}
+ DECIMAL = {"java_class": "BigDecimal.class", "python_class": Decimal, "id": 13}
+ STRUCT = {"java_class": "Void.class", "python_class": None, "id": 14}
+ LIST = {"java_class": "Void.class", "python_class": None, "id": 15}
+ MAP = {"java_class": "Void.class", "python_class": None, "id": 16}
+
+
+class Type(object):
+ length: int
+ scale: int
+ precision: int
+
+ def __init__(self):
+ pass
+
+ def type_id(self):
+ pass
+
+ def is_primitive_type(self):
+ return False
+
+ def as_primitive_type(self):
+ raise ValueError("Not a primitive type: " + self)
+
+ def as_struct_type(self):
+ raise ValueError("Not a struct type: " + self)
+
+ def as_list_type(self):
+ raise ValueError("Not a list type: " + self)
+
+ def asMapType(self):
Review comment:
Looks like this name wasn't updated to snake_case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai removed a comment on pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
samredai removed a comment on pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#issuecomment-948007271
@jun-he do you think you could give me access to this branch `jun-he:jun/add-types` in your forked repo? I have a commit queued up to add test coverage to pass CI and figured we can do that in the same PR. I also added some of the additional linters/autoformatting we discussed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r733331172
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
+ INTEGER = {"java_class": "Integer.class", "python_class": int, "id": 2}
+ LONG = {"java_class": "Long.class", "python_class": int, "id": 3}
+ FLOAT = {"java_class": "Float.class", "python_class": float, "id": 4}
+ DOUBLE = {"java_class": "Double.class", "python_class": float, "id": 5}
+ DATE = {"java_class": "Integer.class", "python_class": int, "id": 6}
+ TIME = {"java_class": "Long.class", "python_class": int, "id": 7}
+ TIMESTAMP = {"java_class": "Long.class", "python_class": int, "id": 8}
+ STRING = {"java_class": "CharSequence.class", "python_class": str, "id": 9}
+ UUID = {"java_class": "java.util.UUID.class", "python_class": uuid.UUID, "id": 10}
+ FIXED = {"java_class": "ByteBuffer.class", "python_class": bytes, "id": 11}
+ BINARY = {"java_class": "ByteBuffer.class", "python_class": bytearray, "id": 12}
+ DECIMAL = {"java_class": "BigDecimal.class", "python_class": Decimal, "id": 13}
+ STRUCT = {"java_class": "Void.class", "python_class": None, "id": 14}
+ LIST = {"java_class": "Void.class", "python_class": None, "id": 15}
+ MAP = {"java_class": "Void.class", "python_class": None, "id": 16}
+
+
+class Type(object):
+ length: int
+ scale: int
+ precision: int
+
+ def __init__(self):
+ pass
+
+ def type_id(self):
+ pass
+
+ def is_primitive_type(self):
+ return False
+
+ def as_primitive_type(self):
+ raise ValueError("Not a primitive type: " + self)
+
+ def as_struct_type(self):
+ raise ValueError("Not a struct type: " + self)
+
+ def as_list_type(self):
+ raise ValueError("Not a list type: " + self)
+
+ def asMapType(self):
+ raise ValueError("Not a map type: " + self)
+
+ def is_nested_type(self):
+ return False
+
+ def is_struct_type(self):
+ return False
+
+ def is_list_type(self):
+ return False
+
+ def is_map_type(self):
+ return False
+
+ def as_nested_type(self):
+ raise ValueError("Not a nested type: " + self)
+
+
+class PrimitiveType(Type):
+
+ def __eq__(self, other):
+ return type(self) == type(other)
+
+ def __ne__(self, other):
+ return not self.__eq__(other)
Review comment:
What I meant is that the default one in python3 is slightly different from this one in the python legacy module.
This implementation in python_legacy only works if `type(self)` implements `__eq__`.
The default one in python 3 is better and can handle other cases. IIUC, it works like
```
def __ne__(self, other):
result = self.__eq__(other)
if result is NotImplemented:
return NotImplemented
return not result
```
So we should remove `__ne__` here if we want to support `!=`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r723864787
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
+ INTEGER = {"java_class": "Integer.class", "python_class": int, "id": 2}
+ LONG = {"java_class": "Long.class", "python_class": int, "id": 3}
+ FLOAT = {"java_class": "Float.class", "python_class": float, "id": 4}
+ DOUBLE = {"java_class": "Double.class", "python_class": float, "id": 5}
+ DATE = {"java_class": "Integer.class", "python_class": int, "id": 6}
+ TIME = {"java_class": "Long.class", "python_class": int, "id": 7}
+ TIMESTAMP = {"java_class": "Long.class", "python_class": int, "id": 8}
+ STRING = {"java_class": "CharSequence.class", "python_class": str, "id": 9}
+ UUID = {"java_class": "java.util.UUID.class", "python_class": uuid.UUID, "id": 10}
+ FIXED = {"java_class": "ByteBuffer.class", "python_class": bytes, "id": 11}
+ BINARY = {"java_class": "ByteBuffer.class", "python_class": bytearray, "id": 12}
+ DECIMAL = {"java_class": "BigDecimal.class", "python_class": Decimal, "id": 13}
+ STRUCT = {"java_class": "Void.class", "python_class": None, "id": 14}
+ LIST = {"java_class": "Void.class", "python_class": None, "id": 15}
+ MAP = {"java_class": "Void.class", "python_class": None, "id": 16}
+
+
+class Type(object):
+ length: int
+ scale: int
+ precision: int
+
+ def __init__(self):
+ pass
+
+ def type_id(self):
+ pass
+
+ def is_primitive_type(self):
+ return False
+
+ def as_primitive_type(self):
+ raise ValueError("Not a primitive type: " + self)
+
+ def as_struct_type(self):
+ raise ValueError("Not a struct type: " + self)
+
+ def as_list_type(self):
+ raise ValueError("Not a list type: " + self)
+
+ def asMapType(self):
+ raise ValueError("Not a map type: " + self)
+
+ def is_nested_type(self):
+ return False
+
+ def is_struct_type(self):
+ return False
+
+ def is_list_type(self):
+ return False
+
+ def is_map_type(self):
+ return False
+
+ def as_nested_type(self):
+ raise ValueError("Not a nested type: " + self)
+
+
+class PrimitiveType(Type):
+
+ def __eq__(self, other):
+ return type(self) == type(other)
+
+ def __ne__(self, other):
+ return not self.__eq__(other)
Review comment:
Seems not completely same.
Based on https://docs.python.org/3/reference/datamodel.html#object.__ne__
```
By default, object implements __eq__() by using is, returning NotImplemented in the case of a false comparison: True if x is y else NotImplemented. For __ne__(), by default it delegates to __eq__() and inverts the result unless it is NotImplemented.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r723422318
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
+ INTEGER = {"java_class": "Integer.class", "python_class": int, "id": 2}
+ LONG = {"java_class": "Long.class", "python_class": int, "id": 3}
+ FLOAT = {"java_class": "Float.class", "python_class": float, "id": 4}
+ DOUBLE = {"java_class": "Double.class", "python_class": float, "id": 5}
+ DATE = {"java_class": "Integer.class", "python_class": int, "id": 6}
+ TIME = {"java_class": "Long.class", "python_class": int, "id": 7}
+ TIMESTAMP = {"java_class": "Long.class", "python_class": int, "id": 8}
+ STRING = {"java_class": "CharSequence.class", "python_class": str, "id": 9}
+ UUID = {"java_class": "java.util.UUID.class", "python_class": uuid.UUID, "id": 10}
+ FIXED = {"java_class": "ByteBuffer.class", "python_class": bytes, "id": 11}
+ BINARY = {"java_class": "ByteBuffer.class", "python_class": bytearray, "id": 12}
+ DECIMAL = {"java_class": "BigDecimal.class", "python_class": Decimal, "id": 13}
+ STRUCT = {"java_class": "Void.class", "python_class": None, "id": 14}
+ LIST = {"java_class": "Void.class", "python_class": None, "id": 15}
+ MAP = {"java_class": "Void.class", "python_class": None, "id": 16}
+
+
+class Type(object):
+ length: int
+ scale: int
+ precision: int
+
+ def __init__(self):
+ pass
+
+ def type_id(self):
+ pass
+
+ def is_primitive_type(self):
+ return False
+
+ def as_primitive_type(self):
+ raise ValueError("Not a primitive type: " + self)
Review comment:
I think these `as_` and `is_` methods are valuable if we have type annotations. Is the plan to add them later?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r723862802
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
+ INTEGER = {"java_class": "Integer.class", "python_class": int, "id": 2}
+ LONG = {"java_class": "Long.class", "python_class": int, "id": 3}
+ FLOAT = {"java_class": "Float.class", "python_class": float, "id": 4}
+ DOUBLE = {"java_class": "Double.class", "python_class": float, "id": 5}
+ DATE = {"java_class": "Integer.class", "python_class": int, "id": 6}
+ TIME = {"java_class": "Long.class", "python_class": int, "id": 7}
+ TIMESTAMP = {"java_class": "Long.class", "python_class": int, "id": 8}
+ STRING = {"java_class": "CharSequence.class", "python_class": str, "id": 9}
+ UUID = {"java_class": "java.util.UUID.class", "python_class": uuid.UUID, "id": 10}
+ FIXED = {"java_class": "ByteBuffer.class", "python_class": bytes, "id": 11}
+ BINARY = {"java_class": "ByteBuffer.class", "python_class": bytearray, "id": 12}
+ DECIMAL = {"java_class": "BigDecimal.class", "python_class": Decimal, "id": 13}
+ STRUCT = {"java_class": "Void.class", "python_class": None, "id": 14}
+ LIST = {"java_class": "Void.class", "python_class": None, "id": 15}
+ MAP = {"java_class": "Void.class", "python_class": None, "id": 16}
+
+
+class Type(object):
+ length: int
+ scale: int
+ precision: int
Review comment:
Agree and they should not be there for all types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r734090984
##########
File path: python/tests/test_types.py
##########
@@ -0,0 +1,85 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from iceberg.types import (BinaryType, BooleanType, DateType, DecimalType, DoubleType, FixedType,
+ FloatType, IntegerType, ListType, LongType, MapType, NestedField, StringType,
+ StructType, TimestampType, TimestamptzType, TimeType, UUIDType)
+import pytest
+
+
+@pytest.mark.parametrize("input_type",
+ [BooleanType, IntegerType, LongType, FloatType, DoubleType, DateType, TimeType,
+ TimestampType, TimestamptzType, StringType, UUIDType, BinaryType])
+def test_repr_primitive_types(input_type):
+ assert input_type == eval(repr(input_type))
Review comment:
Nice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r729900449
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
+ INTEGER = {"java_class": "Integer.class", "python_class": int, "id": 2}
+ LONG = {"java_class": "Long.class", "python_class": int, "id": 3}
+ FLOAT = {"java_class": "Float.class", "python_class": float, "id": 4}
+ DOUBLE = {"java_class": "Double.class", "python_class": float, "id": 5}
+ DATE = {"java_class": "Integer.class", "python_class": int, "id": 6}
+ TIME = {"java_class": "Long.class", "python_class": int, "id": 7}
+ TIMESTAMP = {"java_class": "Long.class", "python_class": int, "id": 8}
+ STRING = {"java_class": "CharSequence.class", "python_class": str, "id": 9}
+ UUID = {"java_class": "java.util.UUID.class", "python_class": uuid.UUID, "id": 10}
+ FIXED = {"java_class": "ByteBuffer.class", "python_class": bytes, "id": 11}
+ BINARY = {"java_class": "ByteBuffer.class", "python_class": bytearray, "id": 12}
+ DECIMAL = {"java_class": "BigDecimal.class", "python_class": Decimal, "id": 13}
+ STRUCT = {"java_class": "Void.class", "python_class": None, "id": 14}
+ LIST = {"java_class": "Void.class", "python_class": None, "id": 15}
+ MAP = {"java_class": "Void.class", "python_class": None, "id": 16}
+
+
+class Type(object):
+ length: int
+ scale: int
+ precision: int
+
+ def __init__(self):
+ pass
+
+ def type_id(self):
+ pass
+
+ def is_primitive_type(self):
+ return False
+
+ def as_primitive_type(self):
+ raise ValueError("Not a primitive type: " + self)
+
+ def as_struct_type(self):
+ raise ValueError("Not a struct type: " + self)
+
+ def as_list_type(self):
+ raise ValueError("Not a list type: " + self)
+
+ def asMapType(self):
+ raise ValueError("Not a map type: " + self)
+
+ def is_nested_type(self):
+ return False
+
+ def is_struct_type(self):
+ return False
+
+ def is_list_type(self):
+ return False
+
+ def is_map_type(self):
+ return False
+
+ def as_nested_type(self):
+ raise ValueError("Not a nested type: " + self)
+
+
+class PrimitiveType(Type):
+
+ def __eq__(self, other):
+ return type(self) == type(other)
+
+ def __ne__(self, other):
+ return not self.__eq__(other)
Review comment:
Are you sure? This sounds like what you're doing here:
> For __ne__(), by default it delegates to __eq__() and inverts the result unless it is NotImplemented.
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self):
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType[{length}]", is_primitive=True)
Review comment:
The repr string should be something that you can paste into Python to re-create the object, so this should produce `FixedType({length})`. That is, it should use parens instead of square brackets.
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self):
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType[{length}]", is_primitive=True)
+ self._length = length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ def precision(self):
Review comment:
+1 for `@property` and annotations.
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self):
+ return self._is_primitive
+
+
+class FixedType(Type):
Review comment:
I think this needs a `@property` method for `length`.
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self):
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType[{length}]", is_primitive=True)
+ self._length = length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ def precision(self):
+ return self._precision
+
+ def scale(self):
+ return self._scale
+
+
+class NestedField(object):
+ def __init__(self, is_optional: bool, field_id: int, name: str, field_type: Type, doc=None):
+ self._is_optional = is_optional
+ self._id = field_id
+ self._name = name
+ self._type = field_type
+ self._doc = doc
+
+ @property
+ def is_optional(self):
+ return self._is_optional
+
+ @property
+ def is_required(self):
+ return not self._is_optional
+
+ @property
+ def field_id(self):
+ return self._id
+
+ @property
+ def type(self):
+ return self._type
+
+ def __repr__(self):
Review comment:
The implementation here should be the one for `__str__`. The `__repr__` implementation should produce a string that is the Python representation.
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self):
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType[{length}]", is_primitive=True)
+ self._length = length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ def precision(self):
+ return self._precision
+
+ def scale(self):
+ return self._scale
+
+
+class NestedField(object):
+ def __init__(self, is_optional: bool, field_id: int, name: str, field_type: Type, doc=None):
+ self._is_optional = is_optional
+ self._id = field_id
+ self._name = name
+ self._type = field_type
+ self._doc = doc
+
+ @property
+ def is_optional(self):
+ return self._is_optional
+
+ @property
+ def is_required(self):
+ return not self._is_optional
+
+ @property
+ def field_id(self):
+ return self._id
+
+ @property
+ def type(self):
+ return self._type
+
+ def __repr__(self):
+ return f"{self._id}: {self._name}: {'optional' if self._is_optional else 'required'} {self._type}" \
+ "" if self._doc is None else f" ({self._doc})"
+
+ def __str__(self):
+ return self.__repr__()
+
+
+class StructType(Type):
+ def __init__(self, fields: list):
+ super().__init__(f"struct<{', '.join(map(str, fields))}>", f"StructType<{', '.join(map(str, fields))}>")
+ self._fields = fields
Review comment:
Accessor method for fields?
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self):
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType[{length}]", is_primitive=True)
+ self._length = length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ def precision(self):
+ return self._precision
+
+ def scale(self):
+ return self._scale
+
+
+class NestedField(object):
+ def __init__(self, is_optional: bool, field_id: int, name: str, field_type: Type, doc=None):
+ self._is_optional = is_optional
+ self._id = field_id
+ self._name = name
+ self._type = field_type
+ self._doc = doc
+
+ @property
+ def is_optional(self):
+ return self._is_optional
+
+ @property
+ def is_required(self):
+ return not self._is_optional
+
+ @property
+ def field_id(self):
+ return self._id
+
+ @property
+ def type(self):
+ return self._type
+
+ def __repr__(self):
+ return f"{self._id}: {self._name}: {'optional' if self._is_optional else 'required'} {self._type}" \
+ "" if self._doc is None else f" ({self._doc})"
+
+ def __str__(self):
+ return self.__repr__()
+
+
+class StructType(Type):
+ def __init__(self, fields: list):
+ super().__init__(f"struct<{', '.join(map(str, fields))}>", f"StructType<{', '.join(map(str, fields))}>")
+ self._fields = fields
+
+
+class ListType(Type):
+ def __init__(self, element_field: NestedField):
+ super().__init__(f"list<{element_field.type}>", f"ListType<{element_field.type}>")
+ self._element_field = element_field
Review comment:
Accessor method for element?
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self):
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType[{length}]", is_primitive=True)
+ self._length = length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ def precision(self):
+ return self._precision
+
+ def scale(self):
+ return self._scale
+
+
+class NestedField(object):
+ def __init__(self, is_optional: bool, field_id: int, name: str, field_type: Type, doc=None):
+ self._is_optional = is_optional
+ self._id = field_id
+ self._name = name
+ self._type = field_type
+ self._doc = doc
+
+ @property
+ def is_optional(self):
+ return self._is_optional
+
+ @property
+ def is_required(self):
+ return not self._is_optional
+
+ @property
+ def field_id(self):
+ return self._id
+
+ @property
+ def type(self):
+ return self._type
+
+ def __repr__(self):
+ return f"{self._id}: {self._name}: {'optional' if self._is_optional else 'required'} {self._type}" \
+ "" if self._doc is None else f" ({self._doc})"
+
+ def __str__(self):
+ return self.__repr__()
+
+
+class StructType(Type):
+ def __init__(self, fields: list):
+ super().__init__(f"struct<{', '.join(map(str, fields))}>", f"StructType<{', '.join(map(str, fields))}>")
+ self._fields = fields
+
+
+class ListType(Type):
+ def __init__(self, element_field: NestedField):
+ super().__init__(f"list<{element_field.type}>", f"ListType<{element_field.type}>")
+ self._element_field = element_field
+
+
+class MapType(Type):
+ def __init__(self, key_field: NestedField, value_field: NestedField):
+ super().__init__(f"map<{key_field.type}, {value_field.type}>",
+ f"MapType<{key_field.type}, {value_field.type}>")
+ self._key_field = key_field
+ self._value_field = value_field
Review comment:
Accessor methods for key and value?
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self):
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType[{length}]", is_primitive=True)
+ self._length = length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ def precision(self):
+ return self._precision
+
+ def scale(self):
+ return self._scale
+
+
+class NestedField(object):
+ def __init__(self, is_optional: bool, field_id: int, name: str, field_type: Type, doc=None):
+ self._is_optional = is_optional
+ self._id = field_id
+ self._name = name
+ self._type = field_type
+ self._doc = doc
+
+ @property
+ def is_optional(self):
+ return self._is_optional
+
+ @property
+ def is_required(self):
+ return not self._is_optional
+
+ @property
+ def field_id(self):
+ return self._id
+
+ @property
+ def type(self):
+ return self._type
+
+ def __repr__(self):
+ return f"{self._id}: {self._name}: {'optional' if self._is_optional else 'required'} {self._type}" \
+ "" if self._doc is None else f" ({self._doc})"
+
+ def __str__(self):
+ return self.__repr__()
+
+
+class StructType(Type):
+ def __init__(self, fields: list):
+ super().__init__(f"struct<{', '.join(map(str, fields))}>", f"StructType<{', '.join(map(str, fields))}>")
+ self._fields = fields
Review comment:
The main thing is that fields should be accessible without going directly to `_fields` because convention says that is private.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r723864787
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
+ INTEGER = {"java_class": "Integer.class", "python_class": int, "id": 2}
+ LONG = {"java_class": "Long.class", "python_class": int, "id": 3}
+ FLOAT = {"java_class": "Float.class", "python_class": float, "id": 4}
+ DOUBLE = {"java_class": "Double.class", "python_class": float, "id": 5}
+ DATE = {"java_class": "Integer.class", "python_class": int, "id": 6}
+ TIME = {"java_class": "Long.class", "python_class": int, "id": 7}
+ TIMESTAMP = {"java_class": "Long.class", "python_class": int, "id": 8}
+ STRING = {"java_class": "CharSequence.class", "python_class": str, "id": 9}
+ UUID = {"java_class": "java.util.UUID.class", "python_class": uuid.UUID, "id": 10}
+ FIXED = {"java_class": "ByteBuffer.class", "python_class": bytes, "id": 11}
+ BINARY = {"java_class": "ByteBuffer.class", "python_class": bytearray, "id": 12}
+ DECIMAL = {"java_class": "BigDecimal.class", "python_class": Decimal, "id": 13}
+ STRUCT = {"java_class": "Void.class", "python_class": None, "id": 14}
+ LIST = {"java_class": "Void.class", "python_class": None, "id": 15}
+ MAP = {"java_class": "Void.class", "python_class": None, "id": 16}
+
+
+class Type(object):
+ length: int
+ scale: int
+ precision: int
+
+ def __init__(self):
+ pass
+
+ def type_id(self):
+ pass
+
+ def is_primitive_type(self):
+ return False
+
+ def as_primitive_type(self):
+ raise ValueError("Not a primitive type: " + self)
+
+ def as_struct_type(self):
+ raise ValueError("Not a struct type: " + self)
+
+ def as_list_type(self):
+ raise ValueError("Not a list type: " + self)
+
+ def asMapType(self):
+ raise ValueError("Not a map type: " + self)
+
+ def is_nested_type(self):
+ return False
+
+ def is_struct_type(self):
+ return False
+
+ def is_list_type(self):
+ return False
+
+ def is_map_type(self):
+ return False
+
+ def as_nested_type(self):
+ raise ValueError("Not a nested type: " + self)
+
+
+class PrimitiveType(Type):
+
+ def __eq__(self, other):
+ return type(self) == type(other)
+
+ def __ne__(self, other):
+ return not self.__eq__(other)
Review comment:
Seems not completely same.
Based on https://docs.python.org/3/reference/datamodel.html#object.__ne__
```
By default, object implements __eq__() by using is, returning NotImplemented in the case of a false comparison: True if x is y else NotImplemented. For __ne__(), by default it delegates to __eq__() and inverts the result unless it is NotImplemented.
```
default will throw `NotImplemented` if false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r723420143
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
+ INTEGER = {"java_class": "Integer.class", "python_class": int, "id": 2}
+ LONG = {"java_class": "Long.class", "python_class": int, "id": 3}
+ FLOAT = {"java_class": "Float.class", "python_class": float, "id": 4}
+ DOUBLE = {"java_class": "Double.class", "python_class": float, "id": 5}
+ DATE = {"java_class": "Integer.class", "python_class": int, "id": 6}
+ TIME = {"java_class": "Long.class", "python_class": int, "id": 7}
+ TIMESTAMP = {"java_class": "Long.class", "python_class": int, "id": 8}
+ STRING = {"java_class": "CharSequence.class", "python_class": str, "id": 9}
+ UUID = {"java_class": "java.util.UUID.class", "python_class": uuid.UUID, "id": 10}
+ FIXED = {"java_class": "ByteBuffer.class", "python_class": bytes, "id": 11}
+ BINARY = {"java_class": "ByteBuffer.class", "python_class": bytearray, "id": 12}
+ DECIMAL = {"java_class": "BigDecimal.class", "python_class": Decimal, "id": 13}
+ STRUCT = {"java_class": "Void.class", "python_class": None, "id": 14}
+ LIST = {"java_class": "Void.class", "python_class": None, "id": 15}
+ MAP = {"java_class": "Void.class", "python_class": None, "id": 16}
+
+
+class Type(object):
+ length: int
+ scale: int
+ precision: int
+
+ def __init__(self):
+ pass
+
+ def type_id(self):
+ pass
+
+ def is_primitive_type(self):
+ return False
+
+ def as_primitive_type(self):
+ raise ValueError("Not a primitive type: " + self)
+
+ def as_struct_type(self):
+ raise ValueError("Not a struct type: " + self)
+
+ def as_list_type(self):
+ raise ValueError("Not a list type: " + self)
+
+ def asMapType(self):
+ raise ValueError("Not a map type: " + self)
+
+ def is_nested_type(self):
+ return False
+
+ def is_struct_type(self):
+ return False
+
+ def is_list_type(self):
+ return False
+
+ def is_map_type(self):
+ return False
+
+ def as_nested_type(self):
+ raise ValueError("Not a nested type: " + self)
+
+
+class PrimitiveType(Type):
+
+ def __eq__(self, other):
+ return type(self) == type(other)
+
+ def __ne__(self, other):
+ return not self.__eq__(other)
Review comment:
This isn't the default implementation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r733263811
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,824 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from decimal import Decimal
+from enum import Enum, unique
+import uuid
+
+
+@unique
+class TypeID(Enum):
+ BOOLEAN = {"java_class": "Boolean.class", "python_class": bool, "id": 1}
+ INTEGER = {"java_class": "Integer.class", "python_class": int, "id": 2}
+ LONG = {"java_class": "Long.class", "python_class": int, "id": 3}
+ FLOAT = {"java_class": "Float.class", "python_class": float, "id": 4}
+ DOUBLE = {"java_class": "Double.class", "python_class": float, "id": 5}
+ DATE = {"java_class": "Integer.class", "python_class": int, "id": 6}
+ TIME = {"java_class": "Long.class", "python_class": int, "id": 7}
+ TIMESTAMP = {"java_class": "Long.class", "python_class": int, "id": 8}
+ STRING = {"java_class": "CharSequence.class", "python_class": str, "id": 9}
+ UUID = {"java_class": "java.util.UUID.class", "python_class": uuid.UUID, "id": 10}
+ FIXED = {"java_class": "ByteBuffer.class", "python_class": bytes, "id": 11}
+ BINARY = {"java_class": "ByteBuffer.class", "python_class": bytearray, "id": 12}
+ DECIMAL = {"java_class": "BigDecimal.class", "python_class": Decimal, "id": 13}
+ STRUCT = {"java_class": "Void.class", "python_class": None, "id": 14}
+ LIST = {"java_class": "Void.class", "python_class": None, "id": 15}
+ MAP = {"java_class": "Void.class", "python_class": None, "id": 16}
+
+
+class Type(object):
+ length: int
+ scale: int
+ precision: int
+
+ def __init__(self):
+ pass
+
+ def type_id(self):
+ pass
+
+ def is_primitive_type(self):
+ return False
+
+ def as_primitive_type(self):
+ raise ValueError("Not a primitive type: " + self)
Review comment:
Currently, I didn't add it and might add it later if needed.
Usually, `as_type` is used to do the real data transform in python and not sure a Java `as_type` kind of method (just return the concrete class type) is helpful in python.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r733266480
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self):
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType[{length}]", is_primitive=True)
+ self._length = length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ def precision(self):
+ return self._precision
+
+ def scale(self):
+ return self._scale
+
+
+class NestedField(object):
+ def __init__(self, is_optional: bool, field_id: int, name: str, field_type: Type, doc=None):
+ self._is_optional = is_optional
+ self._id = field_id
+ self._name = name
+ self._type = field_type
+ self._doc = doc
+
+ @property
+ def is_optional(self):
+ return self._is_optional
+
+ @property
+ def is_required(self):
+ return not self._is_optional
+
+ @property
+ def field_id(self):
+ return self._id
+
+ @property
+ def type(self):
+ return self._type
+
+ def __repr__(self):
+ return f"{self._id}: {self._name}: {'optional' if self._is_optional else 'required'} {self._type}" \
+ "" if self._doc is None else f" ({self._doc})"
+
+ def __str__(self):
+ return self.__repr__()
+
+
+class StructType(Type):
+ def __init__(self, fields: list):
+ super().__init__(f"struct<{', '.join(map(str, fields))}>", f"StructType<{', '.join(map(str, fields))}>")
+ self._fields = fields
Review comment:
SG, updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3234: Support all iceberg types in python library
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3234:
URL: https://github.com/apache/iceberg/pull/3234#discussion_r733267160
##########
File path: python/src/iceberg/types.py
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+class Type(object):
+ def __init__(self, type_string: str, repr_string: str, is_primitive=False):
+ self._type_string = type_string
+ self._repr_string = repr_string
+ self._is_primitive = is_primitive
+
+ def __repr__(self):
+ return self._repr_string
+
+ def __str__(self):
+ return self._type_string
+
+ @property
+ def is_primitive(self):
+ return self._is_primitive
+
+
+class FixedType(Type):
+ def __init__(self, length: int):
+ super().__init__(f"fixed[{length}]", f"FixedType[{length}]", is_primitive=True)
+ self._length = length
+
+
+class DecimalType(Type):
+ def __init__(self, precision: int, scale: int):
+ super().__init__(f"decimal({precision}, {scale})", f"DecimalType({precision}, {scale})", is_primitive=True)
+ self._precision = precision
+ self._scale = scale
+
+ def precision(self):
+ return self._precision
+
+ def scale(self):
+ return self._scale
+
+
+class NestedField(object):
+ def __init__(self, is_optional: bool, field_id: int, name: str, field_type: Type, doc=None):
+ self._is_optional = is_optional
+ self._id = field_id
+ self._name = name
+ self._type = field_type
+ self._doc = doc
+
+ @property
+ def is_optional(self):
+ return self._is_optional
+
+ @property
+ def is_required(self):
+ return not self._is_optional
+
+ @property
+ def field_id(self):
+ return self._id
+
+ @property
+ def type(self):
+ return self._type
+
+ def __repr__(self):
+ return f"{self._id}: {self._name}: {'optional' if self._is_optional else 'required'} {self._type}" \
+ "" if self._doc is None else f" ({self._doc})"
+
+ def __str__(self):
+ return self.__repr__()
+
+
+class StructType(Type):
+ def __init__(self, fields: list):
+ super().__init__(f"struct<{', '.join(map(str, fields))}>", f"StructType<{', '.join(map(str, fields))}>")
+ self._fields = fields
+
+
+class ListType(Type):
+ def __init__(self, element_field: NestedField):
+ super().__init__(f"list<{element_field.type}>", f"ListType<{element_field.type}>")
+ self._element_field = element_field
+
+
+class MapType(Type):
+ def __init__(self, key_field: NestedField, value_field: NestedField):
+ super().__init__(f"map<{key_field.type}, {value_field.type}>",
+ f"MapType<{key_field.type}, {value_field.type}>")
+ self._key_field = key_field
+ self._value_field = value_field
Review comment:
Yep, updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org