You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/10/28 02:31:16 UTC
[GitHub] [iceberg] nssalian opened a new pull request #3407: PartitionSpec python API implementation
nssalian opened a new pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407
In this PR:
1. Added the initial implementation of `partition_field`, `schema` and `partition_spec`.
2. Formatted using black.
Most items are pending since Transforms and TypeUtil haven't been implemented in the Python API
CC: @jun-he, @rdblue , @samredai to help review and understand how to proceed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nssalian commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r741557787
##########
File path: python/tests/test_partition_field.py
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from iceberg.partition_field import PartitionField
Review comment:
This sounds like a good idea. Any method that we need to test can test off an instance of the classes we invoke.
partition_field_obj = partitioning.PartitionField()
partition_field_obj.method_to_test() ....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738817683
##########
File path: python/src/iceberg/validation_exception.py
##########
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+class ValidationException(Exception):
+ def __init__(self, message, *args):
+ super.__str__(message, *args)
+
+ def check(self, test: bool, message: str, *args):
Review comment:
`@staticmethod`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738975408
##########
File path: python/src/iceberg/partition_field.py
##########
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+class PartitionField(object):
Review comment:
yep, agree.
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
+ try:
+ return parse.urlencode(input_str, encoding="utf-8")
+ except TypeError as e:
+ raise e
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def _generate_unpartitioned_spec(self):
+ return PartitionSpec(
+ schema=Schema(),
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def unpartitioned(self) -> PartitionSpec:
+ return self._generate_unpartitioned_spec()
+
+ def check_compatibility(self, spec: PartitionSpec, schema: Schema):
+ for field in spec.fields:
+ source_type = schema.find_type(field.source_id)
+ ValidationException().check(
+ source_type != None,
+ f"Cannot find source column for partition field: {field}",
+ )
+ ValidationException().check(
+ source_type.is_primitive,
+ f"Cannot partition by non-primitive source field: {source_type}",
+ )
+ # TODO: Add transform check
+
+ def has_sequential_ids(self, spec: PartitionSpec):
+ index = 0
+ for field in spec.fields:
+ if field.field_id != PARTITION_DATA_ID_START + index:
+ return False
+ index += 1
+ return True
+
+
+class AtomicInteger:
+ # TODO: Move to utils
+ def __init__(self, value=0):
+ self._value = int(value)
+ self._lock = threading.Lock()
+
+ def inc(self, d=1):
+ with self._lock:
+ self._value += int(d)
+ return self._value
+
+ def dec(self, d=1):
+ return self.inc(-d)
+
+ @property
+ def value(self):
+ with self._lock:
+ return self._value
+
+ @value.setter
+ def value(self, v):
+ with self._lock:
+ self._value = int(v)
+ return self._value
+
+
+class Builder(object):
Review comment:
Wondering if we can simplify it to avoid a Java kind of builder class.
Feel the implementation might be further improved, e.g. removing AtomicInteger classes, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738815140
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
+ try:
+ return parse.urlencode(input_str, encoding="utf-8")
+ except TypeError as e:
+ raise e
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def _generate_unpartitioned_spec(self):
Review comment:
Why have a separate method for this? Shouldn't the unpartitioned spec be a constant somewhere in this file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
samredai commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738876550
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
+ try:
+ return parse.urlencode(input_str, encoding="utf-8")
+ except TypeError as e:
+ raise e
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def _generate_unpartitioned_spec(self):
Review comment:
Sorry if I'm off here, but if this is just a helper for getting a commonly used instantiation of this class, shouldn't these just be default arguments to `__init__()` so a user get's this if they just do `p_spec = PartitionSpec()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
samredai commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738876550
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
+ try:
+ return parse.urlencode(input_str, encoding="utf-8")
+ except TypeError as e:
+ raise e
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def _generate_unpartitioned_spec(self):
Review comment:
Sorry if I'm off here, but if this is just a helper for getting a commonly used instantiation of this class, shouldn't these just be default arguments to `__init__()` so a user gets this if they just do `p_spec = PartitionSpec()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nssalian commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r744026167
##########
File path: python/src/iceberg/schema.py
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from .types import StructType, Type, NestedField
+
+
+class Schema(object):
+ alias_to_id: dict = None
+ id_to_field = {}
+ name_to_id: dict = None
+ lowercase_name_to_id: dict = None
+
+ def __init__(self, struct: StructType, schema_id: int, identifier_field_ids: [int]):
+ self._struct = struct
+ self._schema_id = schema_id
+ self._identifier_field_ids = identifier_field_ids
+
+ @property
+ def struct(self):
+ return self._struct
+
+ @property
+ def schema_id(self):
+ return self._schema_id
+
+ @property
+ def identifier_field_ids(self):
+ return self._identifier_field_ids
+
+ def _generate_id_to_field(self):
Review comment:
Agreed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738809482
##########
File path: python/src/iceberg/partition_field.py
##########
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+class PartitionField(object):
+ def __init__(self, source_id: int, field_id: int, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._name = name
+ # TODO: add transform field
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionField):
+ return (
+ self.source_id == other.source_id
+ and self.field_id == other.field_id
+ and self.name == other.name
Review comment:
is it appropriate in Python to use the fields directly, like `self._source_id == other._source_id`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738817378
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
+ try:
+ return parse.urlencode(input_str, encoding="utf-8")
+ except TypeError as e:
+ raise e
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def _generate_unpartitioned_spec(self):
+ return PartitionSpec(
+ schema=Schema(),
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def unpartitioned(self) -> PartitionSpec:
+ return self._generate_unpartitioned_spec()
+
+ def check_compatibility(self, spec: PartitionSpec, schema: Schema):
+ for field in spec.fields:
+ source_type = schema.find_type(field.source_id)
+ ValidationException().check(
+ source_type != None,
+ f"Cannot find source column for partition field: {field}",
+ )
+ ValidationException().check(
+ source_type.is_primitive,
+ f"Cannot partition by non-primitive source field: {source_type}",
+ )
+ # TODO: Add transform check
+
+ def has_sequential_ids(self, spec: PartitionSpec):
+ index = 0
+ for field in spec.fields:
+ if field.field_id != PARTITION_DATA_ID_START + index:
+ return False
+ index += 1
+ return True
+
+
+class AtomicInteger:
+ # TODO: Move to utils
+ def __init__(self, value=0):
+ self._value = int(value)
+ self._lock = threading.Lock()
+
+ def inc(self, d=1):
+ with self._lock:
+ self._value += int(d)
+ return self._value
+
+ def dec(self, d=1):
+ return self.inc(-d)
+
+ @property
+ def value(self):
+ with self._lock:
+ return self._value
+
+ @value.setter
+ def value(self, v):
+ with self._lock:
+ self._value = int(v)
+ return self._value
+
+
+class Builder(object):
+ schema: Schema = None
+ fields: List[PartitionField] = []
+ partition_names = set()
+ dedup_fields = dict()
+ spec_id = 0
+ last_assigned_field_id = AtomicInteger(value=PARTITION_DATA_ID_START - 1)
+
+ def __init__(self, schema: Schema):
+ self._schema = schema
+
+ def builder_for(self, schema):
+ return Builder(schema=schema)
+
+ def next_field_id(self):
+ return self.last_assigned_field_id.inc()
+
+ def check_and_add_partition_name(self, name: str):
+ # return check_and_add_partition_name(name, None)
+ # TODO: needs more schema methods
+ pass
+
+ def check_and_add_partition_name(self, name: str, source_column_id: int):
+ # TODO: needs more schema methods
+ pass
+
+ def check_for_redundant_partitions(self, field: PartitionField):
+ # TODO: needs transforms
+ pass
+
+ def with_spec_id(self, new_spec_id: int) -> Builder:
+ self.spec_id = new_spec_id
+ return self
+
+ def find_source_column(self, source_name: str) -> NestedField:
+ # TODO: needs schema.find_field
+ pass
+
+ def identity(self, source_name: str, target_name: str) -> Builder:
+ # TODO: needs check_for_redundant_partitions
+ pass
+
+ def identity(self, source_name: str) -> Builder:
+ return self.identity(source_name, source_name)
+
+ def year(self, source_name: str, target_name: str) -> Builder:
+ # TODO: needs check_for_redundant_partitions
+ pass
+
+ def year(self, source_name: str) -> Builder:
+ return self.year(source_name, source_name + "_year")
+
+ def month(self, source_name: str, target_name: str) -> Builder:
+ # TODO: needs check_for_redundant_partitions
+ pass
+
+ def month(self, source_name: str) -> Builder:
+ return self.month(source_name, source_name + "_month")
+
+ def day(self, source_name: str, target_name: str) -> Builder:
+ # TODO: needs check_for_redundant_partitions
+ pass
+
+ def day(self, source_name: str) -> Builder:
+ return self.day(source_name, source_name + "_day")
+
+ def hour(self, source_name: str, target_name: str) -> Builder:
+ # TODO: needs check_for_redundant_partitions
+ pass
+
+ def hour(self, source_name: str) -> Builder:
+ return self.hour(source_name, source_name + "_hour")
+
+ def bucket(self, source_name: str, num_buckets: int, target_name: str) -> Builder:
+ # TODO: needs transforms
+ pass
+
+ def bucket(self, source_name: str, num_buckets: int) -> Builder:
+ return self.bucket(source_name, num_buckets, source_name + "_bucket")
+
+ def truncate(self, source_name: str, width: int, target_name: str) -> Builder:
+ # TODO: needs check_and_add_partition_name and transforms
+ pass
+
+ def truncate(self, source_name: str, width: int) -> Builder:
+ return self.truncate(source_name, width, source_name + "_trunc")
+
+ def always_null(self, source_name: str, target_name: str) -> Builder:
+ # TODO: needs check_and_add_partition_name and transforms
+ pass
+
+ def always_null(self, source_name: str) -> Builder:
+ return self.always_null(source_name, source_name + "_null")
+
+ def add(self, source_id: int, field_id: int):
+ # TODO: needs transforms
+ # TODO: Two more add methods are needed
+ pass
+
+ def build(self) -> PartitionSpec:
+ spec = PartitionSpec(
+ self.schema, self.spec_id, self.fields, self.last_assigned_field_id
+ )
+ PartitionSpec().check_compatibility(spec, self.schema)
Review comment:
This should definitely not create a partition spec just to call `check_compatibility`.
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
+ try:
+ return parse.urlencode(input_str, encoding="utf-8")
+ except TypeError as e:
+ raise e
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def _generate_unpartitioned_spec(self):
+ return PartitionSpec(
+ schema=Schema(),
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def unpartitioned(self) -> PartitionSpec:
+ return self._generate_unpartitioned_spec()
+
+ def check_compatibility(self, spec: PartitionSpec, schema: Schema):
+ for field in spec.fields:
+ source_type = schema.find_type(field.source_id)
+ ValidationException().check(
+ source_type != None,
+ f"Cannot find source column for partition field: {field}",
+ )
+ ValidationException().check(
+ source_type.is_primitive,
+ f"Cannot partition by non-primitive source field: {source_type}",
+ )
+ # TODO: Add transform check
+
+ def has_sequential_ids(self, spec: PartitionSpec):
+ index = 0
+ for field in spec.fields:
+ if field.field_id != PARTITION_DATA_ID_START + index:
+ return False
+ index += 1
+ return True
+
+
+class AtomicInteger:
+ # TODO: Move to utils
+ def __init__(self, value=0):
+ self._value = int(value)
+ self._lock = threading.Lock()
+
+ def inc(self, d=1):
+ with self._lock:
+ self._value += int(d)
+ return self._value
+
+ def dec(self, d=1):
+ return self.inc(-d)
+
+ @property
+ def value(self):
+ with self._lock:
+ return self._value
+
+ @value.setter
+ def value(self, v):
+ with self._lock:
+ self._value = int(v)
+ return self._value
+
+
+class Builder(object):
+ schema: Schema = None
+ fields: List[PartitionField] = []
+ partition_names = set()
+ dedup_fields = dict()
+ spec_id = 0
+ last_assigned_field_id = AtomicInteger(value=PARTITION_DATA_ID_START - 1)
+
+ def __init__(self, schema: Schema):
+ self._schema = schema
+
+ def builder_for(self, schema):
+ return Builder(schema=schema)
+
+ def next_field_id(self):
+ return self.last_assigned_field_id.inc()
+
+ def check_and_add_partition_name(self, name: str):
+ # return check_and_add_partition_name(name, None)
+ # TODO: needs more schema methods
+ pass
+
+ def check_and_add_partition_name(self, name: str, source_column_id: int):
+ # TODO: needs more schema methods
+ pass
+
+ def check_for_redundant_partitions(self, field: PartitionField):
+ # TODO: needs transforms
+ pass
+
+ def with_spec_id(self, new_spec_id: int) -> Builder:
+ self.spec_id = new_spec_id
+ return self
+
+ def find_source_column(self, source_name: str) -> NestedField:
+ # TODO: needs schema.find_field
+ pass
+
+ def identity(self, source_name: str, target_name: str) -> Builder:
+ # TODO: needs check_for_redundant_partitions
+ pass
+
+ def identity(self, source_name: str) -> Builder:
+ return self.identity(source_name, source_name)
+
+ def year(self, source_name: str, target_name: str) -> Builder:
+ # TODO: needs check_for_redundant_partitions
+ pass
+
+ def year(self, source_name: str) -> Builder:
+ return self.year(source_name, source_name + "_year")
+
+ def month(self, source_name: str, target_name: str) -> Builder:
+ # TODO: needs check_for_redundant_partitions
+ pass
+
+ def month(self, source_name: str) -> Builder:
+ return self.month(source_name, source_name + "_month")
+
+ def day(self, source_name: str, target_name: str) -> Builder:
+ # TODO: needs check_for_redundant_partitions
+ pass
+
+ def day(self, source_name: str) -> Builder:
+ return self.day(source_name, source_name + "_day")
+
+ def hour(self, source_name: str, target_name: str) -> Builder:
+ # TODO: needs check_for_redundant_partitions
+ pass
+
+ def hour(self, source_name: str) -> Builder:
+ return self.hour(source_name, source_name + "_hour")
+
+ def bucket(self, source_name: str, num_buckets: int, target_name: str) -> Builder:
+ # TODO: needs transforms
+ pass
+
+ def bucket(self, source_name: str, num_buckets: int) -> Builder:
+ return self.bucket(source_name, num_buckets, source_name + "_bucket")
+
+ def truncate(self, source_name: str, width: int, target_name: str) -> Builder:
+ # TODO: needs check_and_add_partition_name and transforms
+ pass
+
+ def truncate(self, source_name: str, width: int) -> Builder:
+ return self.truncate(source_name, width, source_name + "_trunc")
+
+ def always_null(self, source_name: str, target_name: str) -> Builder:
+ # TODO: needs check_and_add_partition_name and transforms
+ pass
+
+ def always_null(self, source_name: str) -> Builder:
+ return self.always_null(source_name, source_name + "_null")
+
+ def add(self, source_id: int, field_id: int):
+ # TODO: needs transforms
+ # TODO: Two more add methods are needed
+ pass
+
+ def build(self) -> PartitionSpec:
+ spec = PartitionSpec(
+ self.schema, self.spec_id, self.fields, self.last_assigned_field_id
+ )
+ PartitionSpec().check_compatibility(spec, self.schema)
Review comment:
This should definitely not create a partition spec with only default values just to call `check_compatibility`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738813141
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
Review comment:
Can you help me understand the scope of these? Are the essentially instance variable declarations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nssalian commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r739536028
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
+ try:
+ return parse.urlencode(input_str, encoding="utf-8")
+ except TypeError as e:
+ raise e
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def _generate_unpartitioned_spec(self):
+ return PartitionSpec(
+ schema=Schema(),
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def unpartitioned(self) -> PartitionSpec:
+ return self._generate_unpartitioned_spec()
+
+ def check_compatibility(self, spec: PartitionSpec, schema: Schema):
+ for field in spec.fields:
+ source_type = schema.find_type(field.source_id)
+ ValidationException().check(
+ source_type != None,
+ f"Cannot find source column for partition field: {field}",
+ )
+ ValidationException().check(
+ source_type.is_primitive,
+ f"Cannot partition by non-primitive source field: {source_type}",
+ )
+ # TODO: Add transform check
+
+ def has_sequential_ids(self, spec: PartitionSpec):
+ index = 0
+ for field in spec.fields:
+ if field.field_id != PARTITION_DATA_ID_START + index:
+ return False
+ index += 1
+ return True
+
+
+class AtomicInteger:
+ # TODO: Move to utils
+ def __init__(self, value=0):
+ self._value = int(value)
+ self._lock = threading.Lock()
+
+ def inc(self, d=1):
+ with self._lock:
+ self._value += int(d)
+ return self._value
+
+ def dec(self, d=1):
+ return self.inc(-d)
+
+ @property
+ def value(self):
+ with self._lock:
+ return self._value
+
+ @value.setter
+ def value(self, v):
+ with self._lock:
+ self._value = int(v)
+ return self._value
+
+
+class Builder(object):
Review comment:
Discussion continued here on Slack: https://apache-iceberg.slack.com/archives/C029EE6HQ5D/p1635538598009300
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r739373790
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
+ try:
+ return parse.urlencode(input_str, encoding="utf-8")
+ except TypeError as e:
+ raise e
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def _generate_unpartitioned_spec(self):
+ return PartitionSpec(
+ schema=Schema(),
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def unpartitioned(self) -> PartitionSpec:
+ return self._generate_unpartitioned_spec()
+
+ def check_compatibility(self, spec: PartitionSpec, schema: Schema):
+ for field in spec.fields:
+ source_type = schema.find_type(field.source_id)
+ ValidationException().check(
+ source_type != None,
+ f"Cannot find source column for partition field: {field}",
+ )
+ ValidationException().check(
+ source_type.is_primitive,
+ f"Cannot partition by non-primitive source field: {source_type}",
+ )
+ # TODO: Add transform check
+
+ def has_sequential_ids(self, spec: PartitionSpec):
+ index = 0
+ for field in spec.fields:
+ if field.field_id != PARTITION_DATA_ID_START + index:
+ return False
+ index += 1
+ return True
+
+
+class AtomicInteger:
+ # TODO: Move to utils
+ def __init__(self, value=0):
+ self._value = int(value)
+ self._lock = threading.Lock()
+
+ def inc(self, d=1):
+ with self._lock:
+ self._value += int(d)
+ return self._value
+
+ def dec(self, d=1):
+ return self.inc(-d)
+
+ @property
+ def value(self):
+ with self._lock:
+ return self._value
+
+ @value.setter
+ def value(self, v):
+ with self._lock:
+ self._value = int(v)
+ return self._value
+
+
+class Builder(object):
Review comment:
I like constructing objects in this kind of pattern but I think we can achieve it without using typical Java builder in a more pythonic way. For example,
```
def buildermethod(func):
def wrapper(self, *args, **kwargs):
func(self, *args, **kwargs)
return self
return wrapper
class PartitionSpec:
def __init__(self):
pass
@buildermethod
def with_schema(self, schema):
# additional checks
self_schema = schema
...
partition_spec = PartitionSpec().with_schema(schama)...
```
Also, we won't be able to get immutability that Java builder offers in Python.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
samredai commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738875411
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
Review comment:
This should be a falsey check, `if not self.fields_by_source_id:`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
jun-he commented on pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#issuecomment-1020611756
@nssalian BTW, you may also check if the unit test coverage is good enough to pass the threshold.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r791198968
##########
File path: python/src/iceberg/partitioning.py
##########
@@ -0,0 +1,285 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+from typing import List, Tuple, Dict
+
+from iceberg.types import NestedField, Schema, index_by_id
+from iceberg.exceptions import ValidationException
+
+
+class PartitionField:
+ def __init__(self, source_id: int, field_id: int, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._name = name
+ # TODO: add transform field
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ # TODO: add property for transform
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionField):
+ return (
+ self.source_id == other.source_id
+ and self.field_id == other.field_id
+ and self.name == other.name
+ )
+ return False
+
+ def __str__(self):
+ # TODO: add transform to str representation
+ return f"{self.field_id}: {self.name} ({self._source_id})"
+
+ def __repr__(self):
+ return f"PartitionField(field_id={self.field_id}, name={self.name}, source_id={self._source_id})"
+
+
+def check_compatibility(spec, schema: Schema):
+ for field in spec.fields:
+ source_type = schema.find_type(field.source_id)
+ if source_type is None:
+ raise ValidationException(
+ f"Cannot find source column for partition field: {field}"
+ )
+ if not source_type.is_primitive:
+ raise ValidationException(
+ f"Cannot partition by non-primitive source field: {source_type}"
+ )
+ # TODO: Add transform check
+
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec:
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = tuple(part_fields)
+ self._last_assigned_field_id = last_assigned_field_id
+ self.fields_by_source_id: Dict[int, List[PartitionField]] = {}
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> Tuple[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ if not self.fields_by_source_id:
+ for field in self.fields:
+ self.fields_by_source_id[field.source_id] = [field]
+ return self.fields_by_source_id.get(field_id)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + str(field)
+ partition_spec_str += "]" if len(self.fields) > 0 else "\n"
+ return partition_spec_str
+
+ def __repr__(self):
+ return "PartitionSpec" + self.__str__()
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ # TODO: Need to fix with transforms
+ for field in self.fields:
+ other_field = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ # TODO: Add transform check
+ return False
+ index += 1
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def unpartitioned(self):
+ return PartitionSpec(
+ schema=Schema,
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def has_sequential_ids(self, spec):
+ index = 0
+ for field in spec.fields:
+ if field.field_id != PARTITION_DATA_ID_START + index:
+ return False
+ index += 1
+ return True
+
+
+class Builder:
+ def __init__(self, schema: Schema):
+ self._schema = schema
+ self.fields: List[PartitionField] = []
+ self.partition_names = set()
+ self.dedup_fields = dict()
+ self.spec_id = 0
+ self.last_assigned_field_id = PARTITION_DATA_ID_START - 1
+
+ def builder_for(self, schema):
+ return Builder(schema=schema)
+
+ def next_field_id(self):
Review comment:
add type info for the return result.
##########
File path: python/src/iceberg/partitioning.py
##########
@@ -0,0 +1,285 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+from typing import List, Tuple, Dict
+
+from iceberg.types import NestedField, Schema, index_by_id
+from iceberg.exceptions import ValidationException
+
+
+class PartitionField:
+ def __init__(self, source_id: int, field_id: int, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._name = name
+ # TODO: add transform field
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ # TODO: add property for transform
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionField):
+ return (
+ self.source_id == other.source_id
+ and self.field_id == other.field_id
+ and self.name == other.name
+ )
+ return False
+
+ def __str__(self):
+ # TODO: add transform to str representation
+ return f"{self.field_id}: {self.name} ({self._source_id})"
+
+ def __repr__(self):
+ return f"PartitionField(field_id={self.field_id}, name={self.name}, source_id={self._source_id})"
+
+
+def check_compatibility(spec, schema: Schema):
+ for field in spec.fields:
+ source_type = schema.find_type(field.source_id)
+ if source_type is None:
+ raise ValidationException(
+ f"Cannot find source column for partition field: {field}"
+ )
+ if not source_type.is_primitive:
+ raise ValidationException(
+ f"Cannot partition by non-primitive source field: {source_type}"
+ )
+ # TODO: Add transform check
+
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec:
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = tuple(part_fields)
+ self._last_assigned_field_id = last_assigned_field_id
+ self.fields_by_source_id: Dict[int, List[PartitionField]] = {}
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> Tuple[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ if not self.fields_by_source_id:
+ for field in self.fields:
+ self.fields_by_source_id[field.source_id] = [field]
+ return self.fields_by_source_id.get(field_id)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + str(field)
+ partition_spec_str += "]" if len(self.fields) > 0 else "\n"
+ return partition_spec_str
+
+ def __repr__(self):
+ return "PartitionSpec" + self.__str__()
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ # TODO: Need to fix with transforms
+ for field in self.fields:
+ other_field = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ # TODO: Add transform check
+ return False
+ index += 1
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def unpartitioned(self):
+ return PartitionSpec(
+ schema=Schema,
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def has_sequential_ids(self, spec):
+ index = 0
+ for field in spec.fields:
+ if field.field_id != PARTITION_DATA_ID_START + index:
+ return False
+ index += 1
+ return True
+
+
+class Builder:
+ def __init__(self, schema: Schema):
+ self._schema = schema
+ self.fields: List[PartitionField] = []
+ self.partition_names = set()
+ self.dedup_fields = dict()
+ self.spec_id = 0
+ self.last_assigned_field_id = PARTITION_DATA_ID_START - 1
+
+ def builder_for(self, schema):
Review comment:
nit: add type for schema
##########
File path: python/src/iceberg/partitioning.py
##########
@@ -0,0 +1,285 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+from typing import List, Tuple, Dict
+
+from iceberg.types import NestedField, Schema, index_by_id
+from iceberg.exceptions import ValidationException
+
+
+class PartitionField:
+ def __init__(self, source_id: int, field_id: int, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._name = name
+ # TODO: add transform field
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ # TODO: add property for transform
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionField):
+ return (
+ self.source_id == other.source_id
+ and self.field_id == other.field_id
+ and self.name == other.name
+ )
+ return False
+
+ def __str__(self):
+ # TODO: add transform to str representation
+ return f"{self.field_id}: {self.name} ({self._source_id})"
+
+ def __repr__(self):
+ return f"PartitionField(field_id={self.field_id}, name={self.name}, source_id={self._source_id})"
+
+
+def check_compatibility(spec, schema: Schema):
+ for field in spec.fields:
+ source_type = schema.find_type(field.source_id)
+ if source_type is None:
+ raise ValidationException(
+ f"Cannot find source column for partition field: {field}"
+ )
+ if not source_type.is_primitive:
+ raise ValidationException(
+ f"Cannot partition by non-primitive source field: {source_type}"
+ )
+ # TODO: Add transform check
+
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec:
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = tuple(part_fields)
+ self._last_assigned_field_id = last_assigned_field_id
+ self.fields_by_source_id: Dict[int, List[PartitionField]] = {}
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> Tuple[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ if not self.fields_by_source_id:
+ for field in self.fields:
+ self.fields_by_source_id[field.source_id] = [field]
+ return self.fields_by_source_id.get(field_id)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + str(field)
+ partition_spec_str += "]" if len(self.fields) > 0 else "\n"
+ return partition_spec_str
+
+ def __repr__(self):
+ return "PartitionSpec" + self.__str__()
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ # TODO: Need to fix with transforms
+ for field in self.fields:
+ other_field = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ # TODO: Add transform check
+ return False
+ index += 1
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def unpartitioned(self):
+ return PartitionSpec(
+ schema=Schema,
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def has_sequential_ids(self, spec):
+ index = 0
+ for field in spec.fields:
+ if field.field_id != PARTITION_DATA_ID_START + index:
+ return False
+ index += 1
+ return True
+
+
+class Builder:
Review comment:
Should we make the Builder to be the inner class of PartitionSpec class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
samredai commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r746103955
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
Review comment:
Using `dict` and `list` as generics is only possible in python 3.9+ (added in [PEP 585](https://www.python.org/dev/peps/pep-0585/)).For that to work in older versions, we would have to add `from __future__ import annotations` which doesn't look better imo.
Although I am curious why `defaultdict` is used here instead of a standard `dict` and using `Dict` from typing for the typehints. From what I can see, the default lambda used (which is `list`) is never called in this scenario I see in `_generate_fields_by_source_id()`:
```py
fields_source_to_field_dict = defaultdict(list)
for field in self.fields:
fields_source_to_field_dict[field.source_id] = [field]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738815918
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
+ try:
+ return parse.urlencode(input_str, encoding="utf-8")
+ except TypeError as e:
+ raise e
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def _generate_unpartitioned_spec(self):
+ return PartitionSpec(
+ schema=Schema(),
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def unpartitioned(self) -> PartitionSpec:
+ return self._generate_unpartitioned_spec()
Review comment:
I think this should either be a `@staticmethod` or a method in the package and not a method on `PartitionSpec`. Probably the latter. The only reason why it was a static method on the `PartitionSpec` class in Java is because you can't associate methods with packages directly in Java.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nssalian commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r742463239
##########
File path: python/tests/test_partition_field.py
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from iceberg.partition_field import PartitionField
Review comment:
It looks more cleaner with the class name. ```from iceberg.partitioning import PartitionSpec```. Can be used for init checks and method testing.
I see this pattern in the `test_types` as well and we can keep it the same to be consistent.
##########
File path: python/tests/test_partition_field.py
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from iceberg.partition_field import PartitionField
Review comment:
It looks more cleaner with the class name. ```from iceberg.partitioning import PartitionSpec```. Can be used for init checks and method testing.
I see this pattern in the `test_types` as well and we can keep it the same to be consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nssalian commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r741557787
##########
File path: python/tests/test_partition_field.py
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from iceberg.partition_field import PartitionField
Review comment:
This sounds like a good idea. Any method that we need to test can test off an instance of the classes we invoke.
partition_field_obj = partitioning.PartitionField()
partition_field_obj.method_to_test() ....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nssalian commented on pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
nssalian commented on pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#issuecomment-954897368
Thanks for the review. I'll address the comments in upcoming commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738820332
##########
File path: python/src/iceberg/schema.py
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from .types import StructType, Type, NestedField
+
+
+class Schema(object):
+ alias_to_id: dict = None
+ id_to_field = {}
+ name_to_id: dict = None
+ lowercase_name_to_id: dict = None
+
+ def __init__(self, struct: StructType, schema_id: int, identifier_field_ids: [int]):
+ self._struct = struct
+ self._schema_id = schema_id
+ self._identifier_field_ids = identifier_field_ids
+
+ @property
+ def struct(self):
+ return self._struct
+
+ @property
+ def schema_id(self):
+ return self._schema_id
+
+ @property
+ def identifier_field_ids(self):
+ return self._identifier_field_ids
+
+ def _generate_id_to_field(self):
+ if self.id_to_field is None:
+ index = 0
+ temp_id_to_field = {}
+ for field in self.struct.fields:
+ temp_id_to_field[index] = field
+ index += 1
+ self.id_to_field = temp_id_to_field
Review comment:
The implementation here isn't correct. This should produce a map of field IDs to fields, not from positions to fields. Here's [the implementation](https://github.com/apache/iceberg/blob/fa9bb09055faadf37a0e547b6c1bdd4a8bac0be3/api/src/main/java/org/apache/iceberg/types/IndexById.java#L27).
I think to correctly produce a `Schema` class, we will first need to add the type visitors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738817840
##########
File path: python/src/iceberg/validation_exception.py
##########
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+class ValidationException(Exception):
Review comment:
I think we should probably start an `exceptions.py` file for exceptions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738818582
##########
File path: python/src/iceberg/schema.py
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from .types import StructType, Type, NestedField
+
+
+class Schema(object):
+ alias_to_id: dict = None
+ id_to_field = {}
+ name_to_id: dict = None
+ lowercase_name_to_id: dict = None
+
+ def __init__(self, struct: StructType, schema_id: int, identifier_field_ids: [int]):
+ self._struct = struct
+ self._schema_id = schema_id
+ self._identifier_field_ids = identifier_field_ids
+
+ @property
+ def struct(self):
+ return self._struct
+
+ @property
+ def schema_id(self):
+ return self._schema_id
+
+ @property
+ def identifier_field_ids(self):
+ return self._identifier_field_ids
+
+ def _generate_id_to_field(self):
Review comment:
In the Java classes, these methods are called `index_` because we're creating a map that indexes fields. Should we follow a similar convention?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r739299904
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
+ try:
+ return parse.urlencode(input_str, encoding="utf-8")
+ except TypeError as e:
+ raise e
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def _generate_unpartitioned_spec(self):
+ return PartitionSpec(
+ schema=Schema(),
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def unpartitioned(self) -> PartitionSpec:
+ return self._generate_unpartitioned_spec()
+
+ def check_compatibility(self, spec: PartitionSpec, schema: Schema):
+ for field in spec.fields:
+ source_type = schema.find_type(field.source_id)
+ ValidationException().check(
+ source_type != None,
+ f"Cannot find source column for partition field: {field}",
+ )
+ ValidationException().check(
+ source_type.is_primitive,
+ f"Cannot partition by non-primitive source field: {source_type}",
+ )
+ # TODO: Add transform check
+
+ def has_sequential_ids(self, spec: PartitionSpec):
+ index = 0
+ for field in spec.fields:
+ if field.field_id != PARTITION_DATA_ID_START + index:
+ return False
+ index += 1
+ return True
+
+
+class AtomicInteger:
+ # TODO: Move to utils
+ def __init__(self, value=0):
+ self._value = int(value)
+ self._lock = threading.Lock()
+
+ def inc(self, d=1):
+ with self._lock:
+ self._value += int(d)
+ return self._value
+
+ def dec(self, d=1):
+ return self.inc(-d)
+
+ @property
+ def value(self):
+ with self._lock:
+ return self._value
+
+ @value.setter
+ def value(self, v):
+ with self._lock:
+ self._value = int(v)
+ return self._value
+
+
+class Builder(object):
Review comment:
I think that the builder translates well to python because you're constructing the object through an API that does checking while you build. This also gives us a good way to build partition specs by passing the builder around. I'd vote to keep it for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
samredai commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738861097
##########
File path: python/src/iceberg/partition_field.py
##########
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+class PartitionField(object):
+ def __init__(self, source_id: int, field_id: int, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._name = name
+ # TODO: add transform field
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionField):
+ return (
+ self.source_id == other.source_id
+ and self.field_id == other.field_id
+ and self.name == other.name
Review comment:
I would say that's appropriate. Using the leading underscore variables within the class definition is normal practice and lets us remove `@property` methods in the future without effecting any of the other methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nssalian commented on pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
nssalian commented on pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#issuecomment-954028466
Tests pass locally for 3.7 and 3.9.5. I'll fix the line nits in upcoming commits, waiting for review comments first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] xinbinhuang commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r744178622
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
Review comment:
Hi I'm new to iceberg but this signature looks confusing. Is `source_id` and `field_id` the same thing?
```suggestion
def get_fields_by_source_id(self, field_id: int) -> Optional[List[PartitionField]]:
return self._generate_fields_by_source_id().get(field_id, None)
```
The returns should be `Optional[List[PartitionField]]` instead if the the method will returns `None`.
I'm confused about what `_generate_fields_by_source_id` tries to achieve. Why do we need to generate `fields_source_to_field_dict` every time it's called when `self.fields_by_source_id is None`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
samredai commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738862754
##########
File path: python/tests/test_partition_field.py
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from iceberg.partition_field import PartitionField
+
+
+def test_partition_field_equality():
+ part_field_1 = PartitionField(1, 2, "test_1")
+ part_field_2 = PartitionField(1, 2, "test_1")
+ assert part_field_1 == part_field_2
Review comment:
Can we shorten this to a one-liner?
```py
assert PartitionField(1, 2, "test_1") == PartitionField(1, 2, "test_1")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738816370
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
+ try:
+ return parse.urlencode(input_str, encoding="utf-8")
+ except TypeError as e:
+ raise e
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def _generate_unpartitioned_spec(self):
+ return PartitionSpec(
+ schema=Schema(),
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def unpartitioned(self) -> PartitionSpec:
+ return self._generate_unpartitioned_spec()
Review comment:
There's also an argument to be made for making this a `@classmethod` because it is a factory method. In that case it would be attached to the class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
samredai commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738875483
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
Review comment:
You can just `return` here, no need for `None`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r739574039
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
Review comment:
nit: new line before it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
jun-he commented on pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#issuecomment-955100804
Also, should we always implement `__repr__` in those classes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] xinbinhuang commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r744178622
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
Review comment:
Hi I'm new to iceberg but this signature looks confusing. Is `source_id` and `field_id` the same thing?
```suggestion
def get_fields_by_source_id(self, field_id: int) -> Optional[List[PartitionField]]:
return self._generate_fields_by_source_id().get(field_id, None)
```
The returns should be `Optional[List[PartitionField]]` instead if the the method will returns `None`.
I think this method may not be correct because `_generate_fields_by_source_id` will only returns `fields_source_to_field_dict` the first it's called, and all `None` afterward because the `if self.fields_by_source_id is None:` check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] xinbinhuang commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r744180241
##########
File path: python/src/iceberg/schema.py
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from .types import StructType, Type, NestedField
+
+
+class Schema(object):
+ alias_to_id: dict = None
+ id_to_field = {}
+ name_to_id: dict = None
+ lowercase_name_to_id: dict = None
Review comment:
It seems that here is declaring mutable variables as class attributes. This can be problematic because class attributes are mutable and persist across the same python session. i.e.
```python
class Spec:
id: str = None
name: str = None
spec1 = Spec()
spec2 = Spec()
assert(spec1.id is None)
# change class variable from an instance
spec2.__class__.id = "spec2_id"
assert(spec1.id == "spec2_id")
assert(Spec.id == "spec2_id")
spec3 = Spec()
assert(spec3.id == "spec2_id")
Spec.id = "class_spec_id"
assert(spec1.id == "class_spec_id")
assert(spec2.id == "class_spec_id")
assert(spec3.id == "class_spec_id")
```
In the above example, an instance can change the class variable and it affects all other current and future instances. This can lead to surprising behavior.
To have instance scope only variables, we can put them inside the init without exposing them to the signature. i.e.
```python
def __init__(self, struct: StructType, schema_id: int, identifier_field_ids: [int]):
self._struct = struct
self._schema_id = schema_id
self._identifier_field_ids = identifier_field_ids
# not initialized by the user
self.alias_to_id: dict = None
self.id_to_field = {}
self.name_to_id: dict = None
self.lowercase_name_to_id: dict = None
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738816127
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
+ try:
+ return parse.urlencode(input_str, encoding="utf-8")
+ except TypeError as e:
+ raise e
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def _generate_unpartitioned_spec(self):
+ return PartitionSpec(
+ schema=Schema(),
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def unpartitioned(self) -> PartitionSpec:
+ return self._generate_unpartitioned_spec()
+
+ def check_compatibility(self, spec: PartitionSpec, schema: Schema):
Review comment:
Same here. This is probably a method in the package, not attached to the class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738817088
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
+ try:
+ return parse.urlencode(input_str, encoding="utf-8")
+ except TypeError as e:
+ raise e
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def _generate_unpartitioned_spec(self):
+ return PartitionSpec(
+ schema=Schema(),
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def unpartitioned(self) -> PartitionSpec:
+ return self._generate_unpartitioned_spec()
+
+ def check_compatibility(self, spec: PartitionSpec, schema: Schema):
+ for field in spec.fields:
+ source_type = schema.find_type(field.source_id)
+ ValidationException().check(
+ source_type != None,
+ f"Cannot find source column for partition field: {field}",
+ )
+ ValidationException().check(
+ source_type.is_primitive,
+ f"Cannot partition by non-primitive source field: {source_type}",
+ )
+ # TODO: Add transform check
+
+ def has_sequential_ids(self, spec: PartitionSpec):
+ index = 0
+ for field in spec.fields:
+ if field.field_id != PARTITION_DATA_ID_START + index:
+ return False
+ index += 1
+ return True
+
+
+class AtomicInteger:
Review comment:
Is this needed? I don't think that the builder will be multi-threaded. Using this in Java is probably for the convenience of having a counter in a final field or something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738820754
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
Review comment:
This looks suspicious to me. We should be careful to mirror what Java is doing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r739573454
##########
File path: python/src/iceberg/partition_field.py
##########
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
Review comment:
not sure if the formatter can catch it.
But better to be consistent with a single new line after the license in all the places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r739573903
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
Review comment:
QQ: are defaultdict and List needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nssalian commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r744346044
##########
File path: python/src/iceberg/schema.py
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from .types import StructType, Type, NestedField
+
+
+class Schema(object):
+ alias_to_id: dict = None
+ id_to_field = {}
+ name_to_id: dict = None
+ lowercase_name_to_id: dict = None
Review comment:
Agreed. This is the approach that makes sense. Fixing in the upcoming commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738976688
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
+ try:
+ return parse.urlencode(input_str, encoding="utf-8")
+ except TypeError as e:
+ raise e
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def _generate_unpartitioned_spec(self):
+ return PartitionSpec(
+ schema=Schema(),
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def unpartitioned(self) -> PartitionSpec:
+ return self._generate_unpartitioned_spec()
+
+ def check_compatibility(self, spec: PartitionSpec, schema: Schema):
+ for field in spec.fields:
+ source_type = schema.find_type(field.source_id)
+ ValidationException().check(
+ source_type != None,
+ f"Cannot find source column for partition field: {field}",
+ )
+ ValidationException().check(
+ source_type.is_primitive,
+ f"Cannot partition by non-primitive source field: {source_type}",
+ )
+ # TODO: Add transform check
+
+ def has_sequential_ids(self, spec: PartitionSpec):
+ index = 0
+ for field in spec.fields:
+ if field.field_id != PARTITION_DATA_ID_START + index:
+ return False
+ index += 1
+ return True
+
+
+class AtomicInteger:
+ # TODO: Move to utils
+ def __init__(self, value=0):
+ self._value = int(value)
+ self._lock = threading.Lock()
+
+ def inc(self, d=1):
+ with self._lock:
+ self._value += int(d)
+ return self._value
+
+ def dec(self, d=1):
+ return self.inc(-d)
+
+ @property
+ def value(self):
+ with self._lock:
+ return self._value
+
+ @value.setter
+ def value(self, v):
+ with self._lock:
+ self._value = int(v)
+ return self._value
+
+
+class Builder(object):
Review comment:
Wondering if we can simplify it to avoid a Java kind of builder class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r739572673
##########
File path: python/src/iceberg/partition_field.py
##########
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+class PartitionField(object):
+ def __init__(self, source_id: int, field_id: int, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._name = name
+ # TODO: add transform field
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionField):
+ return (
+ self.source_id == other.source_id
+ and self.field_id == other.field_id
+ and self.name == other.name
+ )
+ return False
+
+ def __str__(self):
Review comment:
may also add `def __repr__` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r739295450
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
+ try:
+ return parse.urlencode(input_str, encoding="utf-8")
+ except TypeError as e:
+ raise e
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def _generate_unpartitioned_spec(self):
Review comment:
Sam's idea sounds good to me. We probably don't need this in Python at all unless you think there is some reason to have a singleton unpartitioned spec, or if there is value in readability from using `PartitionSpec.unpartitioned()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r739494149
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
+ try:
+ return parse.urlencode(input_str, encoding="utf-8")
+ except TypeError as e:
+ raise e
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def _generate_unpartitioned_spec(self):
+ return PartitionSpec(
+ schema=Schema(),
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def unpartitioned(self) -> PartitionSpec:
+ return self._generate_unpartitioned_spec()
+
+ def check_compatibility(self, spec: PartitionSpec, schema: Schema):
+ for field in spec.fields:
+ source_type = schema.find_type(field.source_id)
+ ValidationException().check(
+ source_type != None,
+ f"Cannot find source column for partition field: {field}",
+ )
+ ValidationException().check(
+ source_type.is_primitive,
+ f"Cannot partition by non-primitive source field: {source_type}",
+ )
+ # TODO: Add transform check
+
+ def has_sequential_ids(self, spec: PartitionSpec):
+ index = 0
+ for field in spec.fields:
+ if field.field_id != PARTITION_DATA_ID_START + index:
+ return False
+ index += 1
+ return True
+
+
+class AtomicInteger:
+ # TODO: Move to utils
+ def __init__(self, value=0):
+ self._value = int(value)
+ self._lock = threading.Lock()
+
+ def inc(self, d=1):
+ with self._lock:
+ self._value += int(d)
+ return self._value
+
+ def dec(self, d=1):
+ return self.inc(-d)
+
+ @property
+ def value(self):
+ with self._lock:
+ return self._value
+
+ @value.setter
+ def value(self, v):
+ with self._lock:
+ self._value = int(v)
+ return self._value
+
+
+class Builder(object):
Review comment:
But we cannot enforce immutability of the original object in python and it is always mutable. So it misses the main benefit of builder pattern in Java. So I am not sure if there is a confusion for python users that the original object is expected to be immutable or not.
A builder class here seems to be a boilerplate class with an extra layer to prevent calling the target object's methods but it won't prevent the field in the target object being mutated.
There are also multiple other places using builder pattern in iceberg. It is better that we keep this consistent (either use or not use builder class). I will start a thread in the slack channel to get more options about it from python community about builder class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nssalian commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r741557787
##########
File path: python/tests/test_partition_field.py
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from iceberg.partition_field import PartitionField
Review comment:
This sounds like a good idea. Any method that we need to test can test off an instance of the classes we invoke.
partition_field_obj = partitioning.PartitionField()
partition_field_obj.method_to_test() ....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
samredai commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738871109
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
Review comment:
`(object)` is the default so this can be changed to `class PartitionSpec:`
##########
File path: python/tests/test_partition_field.py
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from iceberg.partition_field import PartitionField
+
+
+def test_partition_field_equality():
+ part_field_1 = PartitionField(1, 2, "test_1")
+ part_field_2 = PartitionField(1, 2, "test_1")
+ assert part_field_1 == part_field_2
+
+
+def test_partition_field_str():
+ part_field_1 = PartitionField(1, 2, "test_1")
+ assert str(part_field_1) == "2: test_1 (1)"
Review comment:
Same here:
```py
assert str(PartitionField(1, 2, "test_1")) == "2: test_1 (1)"
```
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
Review comment:
`Optional` imported but unused
##########
File path: python/src/iceberg/schema.py
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from .types import StructType, Type, NestedField
+
+
+class Schema(object):
+ alias_to_id: dict = None
+ id_to_field = {}
+ name_to_id: dict = None
+ lowercase_name_to_id: dict = None
+
+ def __init__(self, struct: StructType, schema_id: int, identifier_field_ids: [int]):
+ self._struct = struct
+ self._schema_id = schema_id
+ self._identifier_field_ids = identifier_field_ids
+
+ @property
+ def struct(self):
+ return self._struct
+
+ @property
+ def schema_id(self):
+ return self._schema_id
+
+ @property
+ def identifier_field_ids(self):
+ return self._identifier_field_ids
+
+ def _generate_id_to_field(self):
+ if self.id_to_field is None:
+ index = 0
+ temp_id_to_field = {}
+ for field in self.struct.fields:
+ temp_id_to_field[index] = field
+ index += 1
+ self.id_to_field = temp_id_to_field
+ return self.id_to_field
+
+ def find_type(self, field_id):
+ field: NestedField = self._generate_id_to_field().get(field_id)
Review comment:
Instead of type hinting here, how about setting a hint for the return value of `_generate_id_to_field()` in the function signature?
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
Review comment:
These aren't needed until an individual instance is initialized right? I think they should be arguments to `__init__()`. A few reasons being that it's unnecessary noise when doing `dir(PartitionSpec)` since you don't expect a user to set/get them at the class scope, and also the `help()` documentation could be misleading as most users will refer to `__init__()` as a comprehensive requirement for initialization and these would be missing there.
##########
File path: python/tests/test_partition_field.py
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from iceberg.partition_field import PartitionField
Review comment:
(incoming opinion) I picked up a style from someone recently to always import the full module being tested so that the tests more obviously present what part of the source code is being tested. I was initially hesitant about the slight increase in verbosity but I've come to find that it is actually super helpful when done consistently throughout.
So what do you think about keeping this `from iceberg import partitioning` (after consolidating the source code into a `partitioning.py` file as suggested) and then using `partitioning.PartitionField` and `partitioning.PartitionSpec` through the tests?
##########
File path: python/src/iceberg/schema.py
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from .types import StructType, Type, NestedField
+
+
+class Schema(object):
+ alias_to_id: dict = None
+ id_to_field = {}
+ name_to_id: dict = None
+ lowercase_name_to_id: dict = None
+
+ def __init__(self, struct: StructType, schema_id: int, identifier_field_ids: [int]):
+ self._struct = struct
+ self._schema_id = schema_id
+ self._identifier_field_ids = identifier_field_ids
+
+ @property
+ def struct(self):
+ return self._struct
+
+ @property
+ def schema_id(self):
+ return self._schema_id
+
+ @property
+ def identifier_field_ids(self):
+ return self._identifier_field_ids
+
+ def _generate_id_to_field(self):
+ if self.id_to_field is None:
Review comment:
Since anything falsey is unacceptable here, maybe change this to
```py
if not self.id_to_field:
```
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
+ try:
+ return parse.urlencode(input_str, encoding="utf-8")
+ except TypeError as e:
+ raise e
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def _generate_unpartitioned_spec(self):
+ return PartitionSpec(
+ schema=Schema(),
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def unpartitioned(self) -> PartitionSpec:
+ return self._generate_unpartitioned_spec()
+
+ def check_compatibility(self, spec: PartitionSpec, schema: Schema):
+ for field in spec.fields:
+ source_type = schema.find_type(field.source_id)
+ ValidationException().check(
+ source_type != None,
+ f"Cannot find source column for partition field: {field}",
+ )
+ ValidationException().check(
+ source_type.is_primitive,
+ f"Cannot partition by non-primitive source field: {source_type}",
+ )
+ # TODO: Add transform check
+
+ def has_sequential_ids(self, spec: PartitionSpec):
+ index = 0
+ for field in spec.fields:
+ if field.field_id != PARTITION_DATA_ID_START + index:
+ return False
+ index += 1
+ return True
+
+
+class AtomicInteger:
+ # TODO: Move to utils
+ def __init__(self, value=0):
+ self._value = int(value)
+ self._lock = threading.Lock()
+
+ def inc(self, d=1):
+ with self._lock:
+ self._value += int(d)
+ return self._value
+
+ def dec(self, d=1):
+ return self.inc(-d)
+
+ @property
+ def value(self):
+ with self._lock:
+ return self._value
+
+ @value.setter
+ def value(self, v):
+ with self._lock:
+ self._value = int(v)
+ return self._value
+
+
+class Builder(object):
+ schema: Schema = None
+ fields: List[PartitionField] = []
+ partition_names = set()
+ dedup_fields = dict()
+ spec_id = 0
+ last_assigned_field_id = AtomicInteger(value=PARTITION_DATA_ID_START - 1)
+
+ def __init__(self, schema: Schema):
+ self._schema = schema
+
+ def builder_for(self, schema):
+ return Builder(schema=schema)
+
+ def next_field_id(self):
+ return self.last_assigned_field_id.inc()
+
+ def check_and_add_partition_name(self, name: str):
+ # return check_and_add_partition_name(name, None)
+ # TODO: needs more schema methods
+ pass
+
+ def check_and_add_partition_name(self, name: str, source_column_id: int):
+ # TODO: needs more schema methods
+ pass
+
+ def check_for_redundant_partitions(self, field: PartitionField):
+ # TODO: needs transforms
+ pass
+
+ def with_spec_id(self, new_spec_id: int) -> Builder:
+ self.spec_id = new_spec_id
+ return self
+
+ def find_source_column(self, source_name: str) -> NestedField:
+ # TODO: needs schema.find_field
+ pass
+
+ def identity(self, source_name: str, target_name: str) -> Builder:
+ # TODO: needs check_for_redundant_partitions
+ pass
+
+ def identity(self, source_name: str) -> Builder:
+ return self.identity(source_name, source_name)
+
+ def year(self, source_name: str, target_name: str) -> Builder:
+ # TODO: needs check_for_redundant_partitions
+ pass
+
+ def year(self, source_name: str) -> Builder:
+ return self.year(source_name, source_name + "_year")
+
+ def month(self, source_name: str, target_name: str) -> Builder:
+ # TODO: needs check_for_redundant_partitions
+ pass
+
+ def month(self, source_name: str) -> Builder:
+ return self.month(source_name, source_name + "_month")
+
+ def day(self, source_name: str, target_name: str) -> Builder:
+ # TODO: needs check_for_redundant_partitions
+ pass
+
+ def day(self, source_name: str) -> Builder:
+ return self.day(source_name, source_name + "_day")
+
+ def hour(self, source_name: str, target_name: str) -> Builder:
+ # TODO: needs check_for_redundant_partitions
+ pass
+
+ def hour(self, source_name: str) -> Builder:
+ return self.hour(source_name, source_name + "_hour")
+
+ def bucket(self, source_name: str, num_buckets: int, target_name: str) -> Builder:
+ # TODO: needs transforms
+ pass
+
+ def bucket(self, source_name: str, num_buckets: int) -> Builder:
+ return self.bucket(source_name, num_buckets, source_name + "_bucket")
+
+ def truncate(self, source_name: str, width: int, target_name: str) -> Builder:
+ # TODO: needs check_and_add_partition_name and transforms
+ pass
+
+ def truncate(self, source_name: str, width: int) -> Builder:
+ return self.truncate(source_name, width, source_name + "_trunc")
+
+ def always_null(self, source_name: str, target_name: str) -> Builder:
+ # TODO: needs check_and_add_partition_name and transforms
+ pass
+
+ def always_null(self, source_name: str) -> Builder:
+ return self.always_null(source_name, source_name + "_null")
+
+ def add(self, source_id: int, field_id: int):
+ # TODO: needs transforms
+ # TODO: Two more add methods are needed
+ pass
+
+ def build(self) -> PartitionSpec:
+ spec = PartitionSpec(
+ self.schema, self.spec_id, self.fields, self.last_assigned_field_id
+ )
+ PartitionSpec().check_compatibility(spec, self.schema)
Review comment:
+1 and I agree that `check_compatibility` should be a function scoped to the package. In general I feel that if `self` is not used then scoping to the package makes more sense.
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
Review comment:
These 3 lines can be reduced to:
```py
partition_spec_str += "]" if len(self.fields) > 0 else "\n"
```
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
Review comment:
Instead of adding typing here, doesn't it make more sense to just add it to the function signature? If `other` is typed to `PartitionSpec` than the type hint for fields is implied by the type hint in the `PartitionSpec` constructor where there is:
```py
...
part_fields: List[PartitionField],
...
```
##########
File path: python/src/iceberg/schema.py
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from .types import StructType, Type, NestedField
+
+
+class Schema(object):
+ alias_to_id: dict = None
+ id_to_field = {}
Review comment:
missing a `dict` type hint
##########
File path: python/src/iceberg/validation_exception.py
##########
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+class ValidationException(Exception):
+ def __init__(self, message, *args):
+ super.__str__(message, *args)
+
+ def check(self, test: bool, message: str, *args):
Review comment:
I'm hesitant to add logic to a custom exception since it isn't obvious what it's doing in the actual location where the check is happening. How about keeping this as:
```py
class ValidationException(Exception):
pass
```
and using it as:
```py
if x != y:
raise ValidationException("custom message here")
```
##########
File path: python/src/iceberg/validation_exception.py
##########
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+class ValidationException(Exception):
Review comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r791198968
##########
File path: python/src/iceberg/partitioning.py
##########
@@ -0,0 +1,285 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+from typing import List, Tuple, Dict
+
+from iceberg.types import NestedField, Schema, index_by_id
+from iceberg.exceptions import ValidationException
+
+
+class PartitionField:
+ def __init__(self, source_id: int, field_id: int, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._name = name
+ # TODO: add transform field
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ # TODO: add property for transform
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionField):
+ return (
+ self.source_id == other.source_id
+ and self.field_id == other.field_id
+ and self.name == other.name
+ )
+ return False
+
+ def __str__(self):
+ # TODO: add transform to str representation
+ return f"{self.field_id}: {self.name} ({self._source_id})"
+
+ def __repr__(self):
+ return f"PartitionField(field_id={self.field_id}, name={self.name}, source_id={self._source_id})"
+
+
+def check_compatibility(spec, schema: Schema):
+ for field in spec.fields:
+ source_type = schema.find_type(field.source_id)
+ if source_type is None:
+ raise ValidationException(
+ f"Cannot find source column for partition field: {field}"
+ )
+ if not source_type.is_primitive:
+ raise ValidationException(
+ f"Cannot partition by non-primitive source field: {source_type}"
+ )
+ # TODO: Add transform check
+
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec:
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = tuple(part_fields)
+ self._last_assigned_field_id = last_assigned_field_id
+ self.fields_by_source_id: Dict[int, List[PartitionField]] = {}
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> Tuple[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ if not self.fields_by_source_id:
+ for field in self.fields:
+ self.fields_by_source_id[field.source_id] = [field]
+ return self.fields_by_source_id.get(field_id)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + str(field)
+ partition_spec_str += "]" if len(self.fields) > 0 else "\n"
+ return partition_spec_str
+
+ def __repr__(self):
+ return "PartitionSpec" + self.__str__()
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ # TODO: Need to fix with transforms
+ for field in self.fields:
+ other_field = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ # TODO: Add transform check
+ return False
+ index += 1
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def unpartitioned(self):
+ return PartitionSpec(
+ schema=Schema,
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def has_sequential_ids(self, spec):
+ index = 0
+ for field in spec.fields:
+ if field.field_id != PARTITION_DATA_ID_START + index:
+ return False
+ index += 1
+ return True
+
+
+class Builder:
+ def __init__(self, schema: Schema):
+ self._schema = schema
+ self.fields: List[PartitionField] = []
+ self.partition_names = set()
+ self.dedup_fields = dict()
+ self.spec_id = 0
+ self.last_assigned_field_id = PARTITION_DATA_ID_START - 1
+
+ def builder_for(self, schema):
+ return Builder(schema=schema)
+
+ def next_field_id(self):
Review comment:
add type info for the return result.
##########
File path: python/src/iceberg/partitioning.py
##########
@@ -0,0 +1,285 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+from typing import List, Tuple, Dict
+
+from iceberg.types import NestedField, Schema, index_by_id
+from iceberg.exceptions import ValidationException
+
+
+class PartitionField:
+ def __init__(self, source_id: int, field_id: int, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._name = name
+ # TODO: add transform field
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ # TODO: add property for transform
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionField):
+ return (
+ self.source_id == other.source_id
+ and self.field_id == other.field_id
+ and self.name == other.name
+ )
+ return False
+
+ def __str__(self):
+ # TODO: add transform to str representation
+ return f"{self.field_id}: {self.name} ({self._source_id})"
+
+ def __repr__(self):
+ return f"PartitionField(field_id={self.field_id}, name={self.name}, source_id={self._source_id})"
+
+
+def check_compatibility(spec, schema: Schema):
+ for field in spec.fields:
+ source_type = schema.find_type(field.source_id)
+ if source_type is None:
+ raise ValidationException(
+ f"Cannot find source column for partition field: {field}"
+ )
+ if not source_type.is_primitive:
+ raise ValidationException(
+ f"Cannot partition by non-primitive source field: {source_type}"
+ )
+ # TODO: Add transform check
+
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec:
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = tuple(part_fields)
+ self._last_assigned_field_id = last_assigned_field_id
+ self.fields_by_source_id: Dict[int, List[PartitionField]] = {}
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> Tuple[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ if not self.fields_by_source_id:
+ for field in self.fields:
+ self.fields_by_source_id[field.source_id] = [field]
+ return self.fields_by_source_id.get(field_id)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + str(field)
+ partition_spec_str += "]" if len(self.fields) > 0 else "\n"
+ return partition_spec_str
+
+ def __repr__(self):
+ return "PartitionSpec" + self.__str__()
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ # TODO: Need to fix with transforms
+ for field in self.fields:
+ other_field = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ # TODO: Add transform check
+ return False
+ index += 1
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def unpartitioned(self):
+ return PartitionSpec(
+ schema=Schema,
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def has_sequential_ids(self, spec):
+ index = 0
+ for field in spec.fields:
+ if field.field_id != PARTITION_DATA_ID_START + index:
+ return False
+ index += 1
+ return True
+
+
+class Builder:
+ def __init__(self, schema: Schema):
+ self._schema = schema
+ self.fields: List[PartitionField] = []
+ self.partition_names = set()
+ self.dedup_fields = dict()
+ self.spec_id = 0
+ self.last_assigned_field_id = PARTITION_DATA_ID_START - 1
+
+ def builder_for(self, schema):
Review comment:
nit: add type for schema
##########
File path: python/src/iceberg/partitioning.py
##########
@@ -0,0 +1,285 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+from typing import List, Tuple, Dict
+
+from iceberg.types import NestedField, Schema, index_by_id
+from iceberg.exceptions import ValidationException
+
+
+class PartitionField:
+ def __init__(self, source_id: int, field_id: int, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._name = name
+ # TODO: add transform field
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ # TODO: add property for transform
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionField):
+ return (
+ self.source_id == other.source_id
+ and self.field_id == other.field_id
+ and self.name == other.name
+ )
+ return False
+
+ def __str__(self):
+ # TODO: add transform to str representation
+ return f"{self.field_id}: {self.name} ({self._source_id})"
+
+ def __repr__(self):
+ return f"PartitionField(field_id={self.field_id}, name={self.name}, source_id={self._source_id})"
+
+
+def check_compatibility(spec, schema: Schema):
+ for field in spec.fields:
+ source_type = schema.find_type(field.source_id)
+ if source_type is None:
+ raise ValidationException(
+ f"Cannot find source column for partition field: {field}"
+ )
+ if not source_type.is_primitive:
+ raise ValidationException(
+ f"Cannot partition by non-primitive source field: {source_type}"
+ )
+ # TODO: Add transform check
+
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec:
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = tuple(part_fields)
+ self._last_assigned_field_id = last_assigned_field_id
+ self.fields_by_source_id: Dict[int, List[PartitionField]] = {}
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> Tuple[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ if not self.fields_by_source_id:
+ for field in self.fields:
+ self.fields_by_source_id[field.source_id] = [field]
+ return self.fields_by_source_id.get(field_id)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + str(field)
+ partition_spec_str += "]" if len(self.fields) > 0 else "\n"
+ return partition_spec_str
+
+ def __repr__(self):
+ return "PartitionSpec" + self.__str__()
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ # TODO: Need to fix with transforms
+ for field in self.fields:
+ other_field = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ # TODO: Add transform check
+ return False
+ index += 1
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def unpartitioned(self):
+ return PartitionSpec(
+ schema=Schema,
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def has_sequential_ids(self, spec):
+ index = 0
+ for field in spec.fields:
+ if field.field_id != PARTITION_DATA_ID_START + index:
+ return False
+ index += 1
+ return True
+
+
+class Builder:
Review comment:
Should we make the Builder to be the inner class of PartitionSpec class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] xinbinhuang commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r744179168
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
Review comment:
Can a single `source_id` map to multiple `fields`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] xinbinhuang commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r744178622
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
Review comment:
Hi I'm new to iceberg but this signature looks confusing. Is `source_id` and `field_id` the same thing?
```suggestion
def get_fields_by_source_id(self, field_id: int) -> Optional[List[PartitionField]]:
return self._generate_fields_by_source_id().get(field_id, None)
```
The returns should be `Optional[List[PartitionField]]` instead if the the method will returns `None`.
I'm confused about what `_generate_fields_by_source_id` tries to achieve. Why do we need to generate `fields_source_to_field_dict` every time it's called?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
jun-he commented on pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#issuecomment-1020611756
@nssalian BTW, you may also check if the unit test coverage is good enough to pass the threshold.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r739297141
##########
File path: python/src/iceberg/validation_exception.py
##########
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+class ValidationException(Exception):
+ def __init__(self, message, *args):
+ super.__str__(message, *args)
+
+ def check(self, test: bool, message: str, *args):
Review comment:
Most of the value of this in Java is embedding the string formatting so you don't have `String.format` class all over the place. Since string interpolation is built in with Python, I think Sam is right. We can just define the exception class rather than using the Precondition style check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r739419039
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
+ try:
+ return parse.urlencode(input_str, encoding="utf-8")
+ except TypeError as e:
+ raise e
+
+ def partition_to_path(self):
+ # TODO: Needs transform
+ pass
+
+ def _generate_unpartitioned_spec(self):
+ return PartitionSpec(
+ schema=Schema(),
+ spec_id=0,
+ part_fields=[],
+ last_assigned_field_id=PARTITION_DATA_ID_START - 1,
+ )
+
+ def unpartitioned(self) -> PartitionSpec:
+ return self._generate_unpartitioned_spec()
+
+ def check_compatibility(self, spec: PartitionSpec, schema: Schema):
+ for field in spec.fields:
+ source_type = schema.find_type(field.source_id)
+ ValidationException().check(
+ source_type != None,
+ f"Cannot find source column for partition field: {field}",
+ )
+ ValidationException().check(
+ source_type.is_primitive,
+ f"Cannot partition by non-primitive source field: {source_type}",
+ )
+ # TODO: Add transform check
+
+ def has_sequential_ids(self, spec: PartitionSpec):
+ index = 0
+ for field in spec.fields:
+ if field.field_id != PARTITION_DATA_ID_START + index:
+ return False
+ index += 1
+ return True
+
+
+class AtomicInteger:
+ # TODO: Move to utils
+ def __init__(self, value=0):
+ self._value = int(value)
+ self._lock = threading.Lock()
+
+ def inc(self, d=1):
+ with self._lock:
+ self._value += int(d)
+ return self._value
+
+ def dec(self, d=1):
+ return self.inc(-d)
+
+ @property
+ def value(self):
+ with self._lock:
+ return self._value
+
+ @value.setter
+ def value(self, v):
+ with self._lock:
+ self._value = int(v)
+ return self._value
+
+
+class Builder(object):
Review comment:
That's not really how builders work. It looks more like refinement, but it modifies the original object. I think that will create confusion.
I'd prefer to stick with the builder API that we have already rather than trying to come up with a new pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nssalian commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r739578571
##########
File path: python/src/iceberg/schema.py
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from .types import StructType, Type, NestedField
+
+
+class Schema(object):
+ alias_to_id: dict = None
+ id_to_field = {}
+ name_to_id: dict = None
+ lowercase_name_to_id: dict = None
Review comment:
I think these are simply class variables that might be used for methods. They don't need to be part of the `__init__` since they aren't initialized externally.
Let me fix the PR with the rest of the changes and see if we need the variables for this iteration of change. I'll remove them if not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r739574039
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
Review comment:
nit: new line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738818156
##########
File path: python/src/iceberg/partition_field.py
##########
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+class PartitionField(object):
Review comment:
I think this should be in a file called `partitioning`, along with `PartitionSpec`. No need for a file per class anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nssalian commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738819388
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
Review comment:
Yes, that's right. Internal to the class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r739572941
##########
File path: python/src/iceberg/partition_field.py
##########
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+class PartitionField(object):
+ def __init__(self, source_id: int, field_id: int, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._name = name
+ # TODO: add transform field
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionField):
+ return (
+ self.source_id == other.source_id
+ and self.field_id == other.field_id
+ and self.name == other.name
Review comment:
Agree, this is internal and should be fine to use fields directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r739575204
##########
File path: python/src/iceberg/schema.py
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from .types import StructType, Type, NestedField
+
+
+class Schema(object):
+ alias_to_id: dict = None
+ id_to_field = {}
+ name_to_id: dict = None
+ lowercase_name_to_id: dict = None
Review comment:
IMO, those 4 fields can be included within `__init__`.
Also, if they are not needed in this change, we might consider to add them in the future when we actually need them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738812198
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
Review comment:
Is the purpose of this a defensive copy? Is so, could we use `tuple(part_fields)` instead so that it is immutable and it can't be modified either directly or when returned by `fields()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r738814180
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
+
+ def __eq__(self, other):
+ if isinstance(other, PartitionSpec):
+ if self.spec_id != other.spec_id:
+ return False
+ return self.fields == other.fields
+ return False
+
+ def __str__(self):
+ partition_spec_str = "["
+ for field in self.fields:
+ partition_spec_str += "\n"
+ partition_spec_str += " " + field
+ if len(self.fields) > 0:
+ partition_spec_str += "\n"
+ partition_spec_str += "]"
+ return partition_spec_str
+
+ def compatible_with(self, other):
+ if self.__eq__(other):
+ return True
+
+ if len(self.fields) != len(other.fields):
+ return False
+
+ index = 0
+ for field in self.fields:
+ other_field: PartitionField = other.fields[index]
+ if (
+ field.source_id != other_field.source_id
+ or field.name != other_field.name
+ ):
+ index += 1
+ # TODO: Add transform check
+ return False
+ return True
+
+ def partition_type(self):
+ struct_fields = []
+ # TODO: Needs transform
+ pass
+
+ def escape(self, input_str):
Review comment:
I don't think that we want to expose this as an instance method. Can we move it to a util package?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
samredai commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r746103955
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
Review comment:
Using `dict` and `list` as generics is only possible in python 3.9+ (added in [PEP 585](https://www.python.org/dev/peps/pep-0585/)).For that to work, we would have to add `from __future__ import annotations` which doesn't look better imo.
Although I am curious why `defaultdict` is used here instead of a standard `dict` and using `Dict` from typing for the typehints. From what I can see, the default lambda used (which is `list`) is never called in this scenario I see in `_generate_fields_by_source_id()`:
```py
fields_source_to_field_dict = defaultdict(list)
for field in self.fields:
fields_source_to_field_dict[field.source_id] = [field]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nssalian commented on pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
nssalian commented on pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#issuecomment-965623092
Added some fixes. Still some more left. Will get to it soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] xinbinhuang commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r744178760
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
Review comment:
Alternatively, you can use `@cached_property` to simplify the `if-else` check and be more pythonic.
```python
@cached_property
def fields_by_source_id(self):
fields_source_to_field_dict = defaultdict(list)
for field in self.fields:
fields_source_to_field_dict[field.source_id] = [field]
return fields_source_to_field_dict
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] xinbinhuang commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r744178760
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
+
+ def __init__(
+ self,
+ schema: Schema,
+ spec_id: int,
+ part_fields: List[PartitionField],
+ last_assigned_field_id: int,
+ ):
+ self._schema = schema
+ self._spec_id = spec_id
+ self._fields = [part_field for part_field in part_fields]
+ self._last_assigned_field_id = last_assigned_field_id
+
+ @property
+ def schema(self) -> Schema:
+ return self._schema
+
+ @property
+ def spec_id(self) -> int:
+ return self._spec_id
+
+ @property
+ def fields(self) -> List[PartitionField]:
+ return self._fields
+
+ @property
+ def last_assigned_field_id(self) -> int:
+ return self._last_assigned_field_id
+
+ def is_partitioned(self):
+ return len(self.fields) < 1
+
+ def _generate_fields_by_source_id(self):
+ if self.fields_by_source_id is None:
+ fields_source_to_field_dict = defaultdict(list)
+ for field in self.fields:
+ fields_source_to_field_dict[field.source_id] = [field]
+ return fields_source_to_field_dict
+ return None
+
+ def get_fields_by_source_id(self, field_id: int) -> List[PartitionField]:
+ return self._generate_fields_by_source_id().get(field_id, None)
Review comment:
Alternatively, you can use `@cached_property` to simplify the `if-else` check and be more pythonic.
```python
@cached_property
def fields_by_source_id(self):
fields_source_to_field_dict = defaultdict(list)
for field in self.fields:
fields_source_to_field_dict[field.source_id] = [field]
return fields_source_to_field_dict
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] xinbinhuang commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r744180241
##########
File path: python/src/iceberg/schema.py
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from .types import StructType, Type, NestedField
+
+
+class Schema(object):
+ alias_to_id: dict = None
+ id_to_field = {}
+ name_to_id: dict = None
+ lowercase_name_to_id: dict = None
Review comment:
It seems that here is declaring mutable variables as class attributes. This can be problematic because class attributes are mutable and persist across the same python session. i.e.
```python
class Spec:
id: str = None
name: str = None
spec1 = Spec()
spec2 = Spec()
assert(spec1.id is None)
# change class variable from an instance
spec2.__class__.id = "spec2_id"
assert(spec1.id == "spec2_id")
assert(Spec.id == "spec2_id")
spec3 = Spec()
assert(spec3.id == "spec2_id")
Spec.id = "class_spec_id"
assert(spec1.id == "class_spec_id")
assert(spec2.id == "class_spec_id")
assert(spec3.id == "class_spec_id")
```
In the above example, an instance can change the class variable and it affects all other current and future instances. This can lead to surprising behavior.
To have instance scope only variables, we can put them inside the init without exposing them to the signature. i.e.
```python
def __init__(self, struct: StructType, schema_id: int, identifier_field_ids: [int]):
self._struct = struct
self._schema_id = schema_id
self._identifier_field_ids = identifier_field_ids
# not initialized by the user externally
self.alias_to_id: dict = None
self.id_to_field = {}
self.name_to_id: dict = None
self.lowercase_name_to_id: dict = None
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] xinbinhuang commented on a change in pull request #3407: [Python] #3228: PartitionSpec python API implementation
Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on a change in pull request #3407:
URL: https://github.com/apache/iceberg/pull/3407#discussion_r744180352
##########
File path: python/src/iceberg/partition_spec.py
##########
@@ -0,0 +1,289 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from urllib import parse
+from collections import defaultdict
+from typing import List, Optional
+
+from iceberg.partition_field import PartitionField
+from iceberg.schema import Schema
+from iceberg.types import Type, NestedField
+from iceberg.validation_exception import ValidationException
+
+PARTITION_DATA_ID_START = 1000
+
+
+class PartitionSpec(object):
+ fields_by_source_id: defaultdict[list] = None
+ field_list: List[PartitionField] = None
Review comment:
Similar thoughts as https://github.com/apache/iceberg/pull/3407#discussion_r744180241
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org