You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/04/19 23:47:57 UTC
[GitHub] [iceberg] dramaticlly opened a new pull request, #4590: Python: Add PartitionField
dramaticlly opened a new pull request, #4590:
URL: https://github.com/apache/iceberg/pull/4590
This is 1st step of reintroduce https://github.com/apache/iceberg/issues/3228
adds `PartitionField` in `iceberg/partitioning.py`
```
from iceberg.partitioning import PartitionField
partition_field = PartitionField(3, 1, None, "id")
partition_field
PartitionField(field_id=1, name=id, transform=None, source_id=3)
```
Purposefully keep the change list small to focus only on PartitionField and once checked in we can look into PartitionSpec next.
CC @samredai @rdblue @jun-he
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dramaticlly commented on pull request #4590: Python: Add PartitionField
Posted by GitBox <gi...@apache.org>.
dramaticlly commented on PR #4590:
URL: https://github.com/apache/iceberg/pull/4590#issuecomment-1105856568
@samredai added hash and moved `partitioning` to the `table` module, if you could take another look :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dramaticlly commented on a diff in pull request #4590: Python: Add PartitionField
Posted by GitBox <gi...@apache.org>.
dramaticlly commented on code in PR #4590:
URL: https://github.com/apache/iceberg/pull/4590#discussion_r853645219
##########
python/src/iceberg/partitioning.py:
##########
@@ -0,0 +1,57 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from iceberg.transforms import Transform
+
+
+class PartitionField:
+ """
+ PartitionField is a single element with name and unique id,
+ It represents how one partition value is derived from the source column via transformation
+ """
Review Comment:
added `Attributes` in docstring as suggested!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dramaticlly commented on a diff in pull request #4590: Python: Add PartitionField
Posted by GitBox <gi...@apache.org>.
dramaticlly commented on code in PR #4590:
URL: https://github.com/apache/iceberg/pull/4590#discussion_r857188800
##########
python/src/iceberg/table/partitioning.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from iceberg.transforms import Transform
+
+
+class PartitionField:
+ """
+ PartitionField is a single element with name and unique id,
+ It represents how one partition value is derived from the source column via transformation
+
+ Attributes:
+ source_id(int): The source column id of table's schema
+ field_id(int): The partition field id across all the table metadata's partition specs
+ transform(Transform): The transform used to produce partition values from source column
+ name(str): The name of this partition field
+ """
+
+ def __init__(self, source_id: int, field_id: int, transform: Transform, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._transform = transform
+ self._name = name
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ @property
+ def transform(self) -> Transform:
+ return self._transform
+
+ def __eq__(self, other):
+ return self.field_id == other.field_id
+
+ def __str__(self):
+ return f"{self.field_id}: {self.name}: {self.transform}: ({self.source_id})"
Review Comment:
removed `: ` after transform to match Java implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4590: Python: Add PartitionField
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4590:
URL: https://github.com/apache/iceberg/pull/4590#discussion_r857177870
##########
python/src/iceberg/table/partitioning.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from iceberg.transforms import Transform
+
+
+class PartitionField:
+ """
+ PartitionField is a single element with name and unique id,
+ It represents how one partition value is derived from the source column via transformation
+
+ Attributes:
+ source_id(int): The source column id of table's schema
+ field_id(int): The partition field id across all the table metadata's partition specs
+ transform(Transform): The transform used to produce partition values from source column
+ name(str): The name of this partition field
+ """
+
+ def __init__(self, source_id: int, field_id: int, transform: Transform, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._transform = transform
+ self._name = name
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ @property
+ def transform(self) -> Transform:
+ return self._transform
+
+ def __eq__(self, other):
+ return self.field_id == other.field_id
+
+ def __str__(self):
+ return f"{self.field_id}: {self.name}: {self.transform}: ({self.source_id})"
+
+ def __repr__(self):
+ return (
+ f"PartitionField(field_id={self.field_id}, name={self.name}, transform={self.transform}, source_id={self.source_id})"
Review Comment:
I think you need to use `repr(self.transform)` to build the repr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4590: Python: Add PartitionField
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4590:
URL: https://github.com/apache/iceberg/pull/4590#discussion_r857741610
##########
python/src/iceberg/partitioning.py:
##########
@@ -0,0 +1,57 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from iceberg.transforms import Transform
+
+
+class PartitionField:
Review Comment:
Sorry I was late to see this, but we don't need `__hash__` implementations unless the objects are being used for keys in dicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #4590: Python: Add PartitionField
Posted by GitBox <gi...@apache.org>.
rdblue merged PR #4590:
URL: https://github.com/apache/iceberg/pull/4590
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4590: Python: Add PartitionField
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4590:
URL: https://github.com/apache/iceberg/pull/4590#discussion_r857177786
##########
python/src/iceberg/table/partitioning.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from iceberg.transforms import Transform
+
+
+class PartitionField:
+ """
+ PartitionField is a single element with name and unique id,
+ It represents how one partition value is derived from the source column via transformation
+
+ Attributes:
+ source_id(int): The source column id of table's schema
+ field_id(int): The partition field id across all the table metadata's partition specs
+ transform(Transform): The transform used to produce partition values from source column
+ name(str): The name of this partition field
+ """
+
+ def __init__(self, source_id: int, field_id: int, transform: Transform, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._transform = transform
+ self._name = name
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ @property
+ def transform(self) -> Transform:
+ return self._transform
+
+ def __eq__(self, other):
+ return self.field_id == other.field_id
Review Comment:
I don't think this is correct. It should be source_id, field_id, transform, and name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4590: Python: Add PartitionField
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4590:
URL: https://github.com/apache/iceberg/pull/4590#discussion_r857177818
##########
python/src/iceberg/table/partitioning.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from iceberg.transforms import Transform
+
+
+class PartitionField:
+ """
+ PartitionField is a single element with name and unique id,
+ It represents how one partition value is derived from the source column via transformation
+
+ Attributes:
+ source_id(int): The source column id of table's schema
+ field_id(int): The partition field id across all the table metadata's partition specs
+ transform(Transform): The transform used to produce partition values from source column
+ name(str): The name of this partition field
+ """
+
+ def __init__(self, source_id: int, field_id: int, transform: Transform, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._transform = transform
+ self._name = name
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ @property
+ def transform(self) -> Transform:
+ return self._transform
+
+ def __eq__(self, other):
+ return self.field_id == other.field_id
+
+ def __str__(self):
+ return f"{self.field_id}: {self.name}: {self.transform}: ({self.source_id})"
Review Comment:
I think this should match the JVM one, which doesn't have `: ` after transform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dramaticlly commented on pull request #4590: Python: Add PartitionField
Posted by GitBox <gi...@apache.org>.
dramaticlly commented on PR #4590:
URL: https://github.com/apache/iceberg/pull/4590#issuecomment-1107934502
@rdblue and @samredai updated to address review feedback, appreciate another look :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4590: Python: Add PartitionField
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4590:
URL: https://github.com/apache/iceberg/pull/4590#discussion_r857177925
##########
python/src/iceberg/table/partitioning.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from iceberg.transforms import Transform
+
+
+class PartitionField:
+ """
+ PartitionField is a single element with name and unique id,
+ It represents how one partition value is derived from the source column via transformation
+
+ Attributes:
+ source_id(int): The source column id of table's schema
+ field_id(int): The partition field id across all the table metadata's partition specs
+ transform(Transform): The transform used to produce partition values from source column
+ name(str): The name of this partition field
+ """
+
+ def __init__(self, source_id: int, field_id: int, transform: Transform, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._transform = transform
+ self._name = name
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ @property
+ def transform(self) -> Transform:
+ return self._transform
+
+ def __eq__(self, other):
+ return self.field_id == other.field_id
+
+ def __str__(self):
+ return f"{self.field_id}: {self.name}: {self.transform}: ({self.source_id})"
+
+ def __repr__(self):
+ return (
+ f"PartitionField(field_id={self.field_id}, name={self.name}, transform={self.transform}, source_id={self.source_id})"
+ )
+
+ def __hash__(self):
+ return hash((self.source_id, self.field_id, self.name, self.transform))
Review Comment:
These aren't going to be used in a dict, so you shouldn't need to override `__hash__`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #4590: Python: Add PartitionField
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #4590:
URL: https://github.com/apache/iceberg/pull/4590#issuecomment-1108707523
Looks great. Thanks, @dramaticlly!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on a diff in pull request #4590: Python: Add PartitionField
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on code in PR #4590:
URL: https://github.com/apache/iceberg/pull/4590#discussion_r855658879
##########
python/src/iceberg/partitioning.py:
##########
@@ -0,0 +1,57 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from iceberg.transforms import Transform
+
+
+class PartitionField:
Review Comment:
Not sure the policy for adding hash across python classes, but maybe it will be useful as PartitionField is part of Iceberg public api ,(example if a user tries to make dicts keyed by partitionSpec?) Java version has it, for comparison.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dramaticlly commented on a diff in pull request #4590: Python: Add PartitionField
Posted by GitBox <gi...@apache.org>.
dramaticlly commented on code in PR #4590:
URL: https://github.com/apache/iceberg/pull/4590#discussion_r857188800
##########
python/src/iceberg/table/partitioning.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from iceberg.transforms import Transform
+
+
+class PartitionField:
+ """
+ PartitionField is a single element with name and unique id,
+ It represents how one partition value is derived from the source column via transformation
+
+ Attributes:
+ source_id(int): The source column id of table's schema
+ field_id(int): The partition field id across all the table metadata's partition specs
+ transform(Transform): The transform used to produce partition values from source column
+ name(str): The name of this partition field
+ """
+
+ def __init__(self, source_id: int, field_id: int, transform: Transform, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._transform = transform
+ self._name = name
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ @property
+ def transform(self) -> Transform:
+ return self._transform
+
+ def __eq__(self, other):
+ return self.field_id == other.field_id
+
+ def __str__(self):
+ return f"{self.field_id}: {self.name}: {self.transform}: ({self.source_id})"
Review Comment:
removed `: ` after transform to match [Java implementation](https://github.com/apache/iceberg/blob/master/api/src/main/java/org/apache/iceberg/PartitionField.java#L72)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dramaticlly commented on a diff in pull request #4590: Python: Add PartitionField
Posted by GitBox <gi...@apache.org>.
dramaticlly commented on code in PR #4590:
URL: https://github.com/apache/iceberg/pull/4590#discussion_r857188749
##########
python/src/iceberg/table/partitioning.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from iceberg.transforms import Transform
+
+
+class PartitionField:
+ """
+ PartitionField is a single element with name and unique id,
+ It represents how one partition value is derived from the source column via transformation
+
+ Attributes:
+ source_id(int): The source column id of table's schema
+ field_id(int): The partition field id across all the table metadata's partition specs
+ transform(Transform): The transform used to produce partition values from source column
+ name(str): The name of this partition field
+ """
+
+ def __init__(self, source_id: int, field_id: int, transform: Transform, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._transform = transform
+ self._name = name
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ @property
+ def transform(self) -> Transform:
+ return self._transform
+
+ def __eq__(self, other):
+ return self.field_id == other.field_id
Review Comment:
Thank you, updated the equality as suggested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dramaticlly commented on a diff in pull request #4590: Python: Add PartitionField
Posted by GitBox <gi...@apache.org>.
dramaticlly commented on code in PR #4590:
URL: https://github.com/apache/iceberg/pull/4590#discussion_r853648670
##########
python/src/iceberg/partitioning.py:
##########
@@ -0,0 +1,57 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from iceberg.transforms import Transform
+
+
+class PartitionField:
Review Comment:
Looks like it's mainly used in list or object array from what I can tell, but please let me know if I should add it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a diff in pull request #4590: Python: Add PartitionField
Posted by GitBox <gi...@apache.org>.
samredai commented on code in PR #4590:
URL: https://github.com/apache/iceberg/pull/4590#discussion_r853612744
##########
python/src/iceberg/partitioning.py:
##########
@@ -0,0 +1,57 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from iceberg.transforms import Transform
+
+
+class PartitionField:
+ """
+ PartitionField is a single element with name and unique id,
+ It represents how one partition value is derived from the source column via transformation
+ """
Review Comment:
```suggestion
Attributes:
source_id(int): The field id of the source field in the spec's table schema
field_id(int): The partition field id across all the table metadata's partition specs
transform(Transform): The transform used to produce partition values from source values
name(str): The name of this partition field
"""
```
##########
python/src/iceberg/partitioning.py:
##########
@@ -0,0 +1,57 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from iceberg.transforms import Transform
+
+
+class PartitionField:
Review Comment:
@rdblue do we need to define a hash method for this class? Maybe something like:
```py
def __hash__(self):
return hash((self.source_id, self.field_id, self.name, self.transform))
```
##########
python/src/iceberg/partitioning.py:
##########
@@ -0,0 +1,57 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from iceberg.transforms import Transform
+
+
+class PartitionField:
+ """
+ PartitionField is a single element with name and unique id,
+ It represents how one partition value is derived from the source column via transformation
+ """
+
+ def __init__(self, source_id: int, field_id: int, transform: Transform, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._transform = transform
+ self._name = name
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ @property
+ def transform(self) -> Transform:
+ return self._transform
+
+ def __eq__(self, other):
+ return self.field_id == other.field_id
+
+ def __str__(self):
+ return f"{self.field_id}: {self.name}: {self.transform}: ({self.source_id})"
+
+ def __repr__(self):
+ return (
+ f"PartitionField(field_id={self.field_id}, name={self.name}, transform={self._transform}, source_id={self.source_id})"
Review Comment:
nit: to be consistent with the other variables in the repr, `self._transform` should be `self.transform`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dramaticlly commented on a diff in pull request #4590: Python: Add PartitionField
Posted by GitBox <gi...@apache.org>.
dramaticlly commented on code in PR #4590:
URL: https://github.com/apache/iceberg/pull/4590#discussion_r857188869
##########
python/src/iceberg/table/partitioning.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from iceberg.transforms import Transform
+
+
+class PartitionField:
+ """
+ PartitionField is a single element with name and unique id,
+ It represents how one partition value is derived from the source column via transformation
+
+ Attributes:
+ source_id(int): The source column id of table's schema
+ field_id(int): The partition field id across all the table metadata's partition specs
+ transform(Transform): The transform used to produce partition values from source column
+ name(str): The name of this partition field
+ """
+
+ def __init__(self, source_id: int, field_id: int, transform: Transform, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._transform = transform
+ self._name = name
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ @property
+ def transform(self) -> Transform:
+ return self._transform
+
+ def __eq__(self, other):
+ return self.field_id == other.field_id
+
+ def __str__(self):
+ return f"{self.field_id}: {self.name}: {self.transform}: ({self.source_id})"
+
+ def __repr__(self):
+ return (
+ f"PartitionField(field_id={self.field_id}, name={self.name}, transform={self.transform}, source_id={self.source_id})"
+ )
+
+ def __hash__(self):
+ return hash((self.source_id, self.field_id, self.name, self.transform))
Review Comment:
removed `__hash__` override
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dramaticlly commented on a diff in pull request #4590: Python: Add PartitionField
Posted by GitBox <gi...@apache.org>.
dramaticlly commented on code in PR #4590:
URL: https://github.com/apache/iceberg/pull/4590#discussion_r857188840
##########
python/src/iceberg/table/partitioning.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from iceberg.transforms import Transform
+
+
+class PartitionField:
+ """
+ PartitionField is a single element with name and unique id,
+ It represents how one partition value is derived from the source column via transformation
+
+ Attributes:
+ source_id(int): The source column id of table's schema
+ field_id(int): The partition field id across all the table metadata's partition specs
+ transform(Transform): The transform used to produce partition values from source column
+ name(str): The name of this partition field
+ """
+
+ def __init__(self, source_id: int, field_id: int, transform: Transform, name: str):
+ self._source_id = source_id
+ self._field_id = field_id
+ self._transform = transform
+ self._name = name
+
+ @property
+ def source_id(self) -> int:
+ return self._source_id
+
+ @property
+ def field_id(self) -> int:
+ return self._field_id
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ @property
+ def transform(self) -> Transform:
+ return self._transform
+
+ def __eq__(self, other):
+ return self.field_id == other.field_id
+
+ def __str__(self):
+ return f"{self.field_id}: {self.name}: {self.transform}: ({self.source_id})"
+
+ def __repr__(self):
+ return (
+ f"PartitionField(field_id={self.field_id}, name={self.name}, transform={self.transform}, source_id={self.source_id})"
Review Comment:
updated to use `repr` for transform
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org