You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by fo...@apache.org on 2023/10/15 13:07:29 UTC
[iceberg-python] branch pyiceberg-0.5.x updated (1ac315b -> c42317f)
This is an automated email from the ASF dual-hosted git repository.
fokko pushed a change to branch pyiceberg-0.5.x
in repository https://gitbox.apache.org/repos/asf/iceberg-python.git
from 1ac315b Python: Fix caching of FileSystem (#8549)
new 6c8ea0e Python: Create HadoopFileSystem from netloc (merge request !1060) (#8596)
new 15a75e3 Add missing root files
new 6332f56 docs: Update how to release (#34)
new 9fd9998 Exclude Pydantic 2.4.0, 2.4.1 (#38)
new 724e391 Fix the TableIdentifier (#44)
new f41962e Docs: Fix repo name and url (#54)
new cbd00e2 Fix column rename doc example to reflect correct API (#59)
new 7e0041f Expression: Fix multiple and/or expressions (#65)
new b026c5d Check for empty responses (#69)
new c42317f Bump version to 0.5.1 (#68)
The 10 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/ISSUE_TEMPLATE/iceberg_bug_report.yml | 28 +++++++
.github/ISSUE_TEMPLATE/iceberg_improvement.yml | 11 +++
.github/ISSUE_TEMPLATE/iceberg_question.yml | 11 +++
dev/entrypoint.sh => .github/dependabot.yml | 19 +++--
.../workflows/license_check.yml | 13 ++--
.github/workflows/python-ci-docs.yml | 56 ++++++++++++++
.asf.yaml => .github/workflows/python-ci.yml | 55 ++++++++------
.../workflows/python-integration.yml | 31 ++++++--
.github/workflows/python-release.yml | 88 ++++++++++++++++++++++
.github/workflows/stale.yml | 49 ++++++++++++
.gitignore | 52 +++++++++++++
.pre-commit-config.yaml | 19 ++---
dev/.rat-excludes | 2 +
mkdocs/docs/api.md | 4 +-
mkdocs/docs/how-to-release.md | 16 ++--
mkdocs/docs/verify-release.md | 17 ++++-
mkdocs/mkdocs.yml | 4 +-
pyiceberg/__init__.py | 2 +-
pyiceberg/catalog/rest.py | 10 ++-
pyiceberg/expressions/parser.py | 4 +-
pyiceberg/io/pyarrow.py | 36 ++++-----
pyiceberg/table/__init__.py | 26 ++++++-
pyproject.toml | 4 +-
tests/catalog/test_base.py | 8 +-
tests/expressions/test_parser.py | 8 ++
tests/io/test_pyarrow.py | 27 ++++---
tests/test_integration.py | 12 +--
27 files changed, 499 insertions(+), 113 deletions(-)
create mode 100644 .github/ISSUE_TEMPLATE/iceberg_bug_report.yml
create mode 100644 .github/ISSUE_TEMPLATE/iceberg_improvement.yml
create mode 100644 .github/ISSUE_TEMPLATE/iceberg_question.yml
copy dev/entrypoint.sh => .github/dependabot.yml (74%)
mode change 100755 => 100644
copy dev/entrypoint.sh => .github/workflows/license_check.yml (83%)
mode change 100755 => 100644
create mode 100644 .github/workflows/python-ci-docs.yml
copy .asf.yaml => .github/workflows/python-ci.yml (51%)
copy dev/entrypoint.sh => .github/workflows/python-integration.yml (58%)
mode change 100755 => 100644
create mode 100644 .github/workflows/python-release.yml
create mode 100644 .github/workflows/stale.yml
create mode 100644 .gitignore
[iceberg-python] 07/10: Fix column rename doc example to reflect correct API (#59)
Posted by fo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fokko pushed a commit to branch pyiceberg-0.5.x
in repository https://gitbox.apache.org/repos/asf/iceberg-python.git
commit cbd00e2384526fe3d0744fb33b10d2b26c84749e
Author: Abhishek <ak...@netflix.com>
AuthorDate: Thu Oct 12 13:49:13 2023 -0700
Fix column rename doc example to reflect correct API (#59)
---
mkdocs/docs/api.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/mkdocs/docs/api.md b/mkdocs/docs/api.md
index 55eadc5..c19efd6 100644
--- a/mkdocs/docs/api.md
+++ b/mkdocs/docs/api.md
@@ -195,9 +195,9 @@ Renaming a field in an Iceberg table is simple:
```python
with table.update_schema() as update:
- update.rename("retries", "num_retries")
+ update.rename_column("retries", "num_retries")
# This will rename `confirmed_by` to `exchange`
- update.rename("properties.confirmed_by", "exchange")
+ update.rename_column("properties.confirmed_by", "exchange")
```
### Move column
[iceberg-python] 06/10: Docs: Fix repo name and url (#54)
Posted by fo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fokko pushed a commit to branch pyiceberg-0.5.x
in repository https://gitbox.apache.org/repos/asf/iceberg-python.git
commit f41962ed3db0d4a5200166dd1eed64cc92db389a
Author: Manu Zhang <Ow...@gmail.com>
AuthorDate: Tue Oct 10 04:20:24 2023 +0800
Docs: Fix repo name and url (#54)
---
mkdocs/mkdocs.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/mkdocs/mkdocs.yml b/mkdocs/mkdocs.yml
index 5f35129..90892ac 100644
--- a/mkdocs/mkdocs.yml
+++ b/mkdocs/mkdocs.yml
@@ -17,8 +17,8 @@
---
site_name: PyIceberg
site_url: https://py.iceberg.apache.org/
-repo_url: "https://github.com/apache/iceberg/tree/master/python"
-repo_name: "apache/iceberg/python"
+repo_url: "https://github.com/apache/iceberg-python"
+repo_name: "apache/iceberg-python"
plugins:
- gen-files:
[iceberg-python] 05/10: Fix the TableIdentifier (#44)
Posted by fo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fokko pushed a commit to branch pyiceberg-0.5.x
in repository https://gitbox.apache.org/repos/asf/iceberg-python.git
commit 724e3911a67ed31e7c24d53c9bed01299b796b4b
Author: Fokko Driesprong <fo...@tabular.io>
AuthorDate: Sat Oct 7 15:16:51 2023 +0200
Fix the TableIdentifier (#44)
* Fix the TableIdentifier
* Make the rest of the tests happy
* Add default properties
---
pyiceberg/catalog/rest.py | 6 +++++-
pyiceberg/table/__init__.py | 26 ++++++++++++++++++++++++--
tests/catalog/test_base.py | 8 +++++---
tests/test_integration.py | 12 +++++++-----
4 files changed, 41 insertions(+), 11 deletions(-)
diff --git a/pyiceberg/catalog/rest.py b/pyiceberg/catalog/rest.py
index 0023e18..ad2c510 100644
--- a/pyiceberg/catalog/rest.py
+++ b/pyiceberg/catalog/rest.py
@@ -62,6 +62,7 @@ from pyiceberg.table import (
CommitTableRequest,
CommitTableResponse,
Table,
+ TableIdentifier,
TableMetadata,
)
from pyiceberg.table.sorting import UNSORTED_SORT_ORDER, SortOrder
@@ -301,7 +302,10 @@ class RestCatalog(Catalog):
# Update URI based on overrides
self.uri = config[URI]
- def _split_identifier_for_path(self, identifier: Union[str, Identifier]) -> Properties:
+ def _split_identifier_for_path(self, identifier: Union[str, Identifier, TableIdentifier]) -> Properties:
+ if isinstance(identifier, TableIdentifier):
+ return {"namespace": NAMESPACE_SEPARATOR.join(identifier.namespace.root[1:]), "table": identifier.name}
+
identifier_tuple = self.identifier_to_tuple(identifier)
if len(identifier_tuple) <= 1:
raise NoSuchTableError(f"Missing namespace or invalid identifier: {'.'.join(identifier_tuple)}")
diff --git a/pyiceberg/table/__init__.py b/pyiceberg/table/__init__.py
index b905c95..9f77dff 100644
--- a/pyiceberg/table/__init__.py
+++ b/pyiceberg/table/__init__.py
@@ -74,6 +74,7 @@ from pyiceberg.table.sorting import SortOrder
from pyiceberg.typedef import (
EMPTY_DICT,
IcebergBaseModel,
+ IcebergRootModel,
Identifier,
KeyDefaultDict,
Properties,
@@ -402,8 +403,25 @@ class AssertDefaultSortOrderId(TableRequirement):
default_sort_order_id: int = Field(..., alias="default-sort-order-id")
+class Namespace(IcebergRootModel[List[str]]):
+ """Reference to one or more levels of a namespace."""
+
+ root: List[str] = Field(
+ ...,
+ description='Reference to one or more levels of a namespace',
+ example=['accounting', 'tax'],
+ )
+
+
+class TableIdentifier(IcebergBaseModel):
+ """Fully Qualified identifier to a table."""
+
+ namespace: Namespace
+ name: str
+
+
class CommitTableRequest(IcebergBaseModel):
- identifier: Identifier = Field()
+ identifier: TableIdentifier = Field()
requirements: Tuple[SerializeAsAny[TableRequirement], ...] = Field(default_factory=tuple)
updates: Tuple[SerializeAsAny[TableUpdate], ...] = Field(default_factory=tuple)
@@ -526,7 +544,11 @@ class Table:
def _do_commit(self, updates: Tuple[TableUpdate, ...], requirements: Tuple[TableRequirement, ...]) -> None:
response = self.catalog._commit_table( # pylint: disable=W0212
- CommitTableRequest(identifier=self.identifier[1:], updates=updates, requirements=requirements)
+ CommitTableRequest(
+ identifier=TableIdentifier(namespace=self.identifier[:-1], name=self.identifier[-1]),
+ updates=updates,
+ requirements=requirements,
+ )
) # pylint: disable=W0212
self.metadata = response.metadata
self.metadata_location = response.metadata_location
diff --git a/tests/catalog/test_base.py b/tests/catalog/test_base.py
index d1d7d63..fbec2cb 100644
--- a/tests/catalog/test_base.py
+++ b/tests/catalog/test_base.py
@@ -46,8 +46,10 @@ from pyiceberg.table import (
AddSchemaUpdate,
CommitTableRequest,
CommitTableResponse,
+ Namespace,
SetCurrentSchemaUpdate,
Table,
+ TableIdentifier,
)
from pyiceberg.table.metadata import TableMetadata, TableMetadataV1, new_table_metadata
from pyiceberg.table.sorting import UNSORTED_SORT_ORDER, SortOrder
@@ -119,8 +121,8 @@ class InMemoryCatalog(Catalog):
for update in table_request.updates:
if isinstance(update, AddSchemaUpdate):
add_schema_update: AddSchemaUpdate = update
- identifier = Catalog.identifier_to_tuple(table_request.identifier)
- table = self.__tables[("com", *identifier)]
+ identifier = tuple(table_request.identifier.namespace.root) + (table_request.identifier.name,)
+ table = self.__tables[identifier]
new_metadata = new_table_metadata(
add_schema_update.schema_,
table.metadata.partition_specs[0],
@@ -528,7 +530,7 @@ def test_commit_table(catalog: InMemoryCatalog) -> None:
# When
response = given_table.catalog._commit_table( # pylint: disable=W0212
CommitTableRequest(
- identifier=given_table.identifier[1:],
+ identifier=TableIdentifier(namespace=Namespace(given_table.identifier[:-1]), name=given_table.identifier[-1]),
updates=[
AddSchemaUpdate(schema=new_schema, last_column_id=new_schema.highest_field_id),
SetCurrentSchemaUpdate(schema_id=-1),
diff --git a/tests/test_integration.py b/tests/test_integration.py
index 297749b..6d777cf 100644
--- a/tests/test_integration.py
+++ b/tests/test_integration.py
@@ -46,6 +46,8 @@ from pyiceberg.types import (
TimestampType,
)
+DEFAULT_PROPERTIES = {'write.parquet.compression-codec': 'zstd'}
+
@pytest.fixture()
def catalog() -> Catalog:
@@ -104,25 +106,25 @@ def table(catalog: Catalog) -> Table:
@pytest.mark.integration
def test_table_properties(table: Table) -> None:
- assert table.properties == {}
+ assert table.properties == DEFAULT_PROPERTIES
with table.transaction() as transaction:
transaction.set_properties(abc="🤪")
- assert table.properties == {"abc": "🤪"}
+ assert table.properties == dict(**{"abc": "🤪"}, **DEFAULT_PROPERTIES)
with table.transaction() as transaction:
transaction.remove_properties("abc")
- assert table.properties == {}
+ assert table.properties == DEFAULT_PROPERTIES
table = table.transaction().set_properties(abc="def").commit_transaction()
- assert table.properties == {"abc": "def"}
+ assert table.properties == dict(**{"abc": "def"}, **DEFAULT_PROPERTIES)
table = table.transaction().remove_properties("abc").commit_transaction()
- assert table.properties == {}
+ assert table.properties == DEFAULT_PROPERTIES
@pytest.fixture()
[iceberg-python] 09/10: Check for empty responses (#69)
Posted by fo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fokko pushed a commit to branch pyiceberg-0.5.x
in repository https://gitbox.apache.org/repos/asf/iceberg-python.git
commit b026c5d4b7ba5244f67733870c06c952923e6852
Author: Fokko Driesprong <fo...@tabular.io>
AuthorDate: Sat Oct 14 11:39:04 2023 +0200
Check for empty responses (#69)
---
pyiceberg/catalog/rest.py | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/pyiceberg/catalog/rest.py b/pyiceberg/catalog/rest.py
index ad2c510..20875d3 100644
--- a/pyiceberg/catalog/rest.py
+++ b/pyiceberg/catalog/rest.py
@@ -319,6 +319,10 @@ class RestCatalog(Catalog):
def _handle_non_200_response(self, exc: HTTPError, error_handler: Dict[int, Type[Exception]]) -> None:
exception: Type[Exception]
+
+ if exc.response is None:
+ raise ValueError("Did not receive a response")
+
code = exc.response.status_code
if code in error_handler:
exception = error_handler[code]
[iceberg-python] 02/10: Add missing root files
Posted by fo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fokko pushed a commit to branch pyiceberg-0.5.x
in repository https://gitbox.apache.org/repos/asf/iceberg-python.git
commit 15a75e3e2a403d71876cde35806d6b3283940729
Author: Fokko Driesprong <fo...@tabular.io>
AuthorDate: Fri Sep 29 14:10:46 2023 +0200
Add missing root files
---
.github/ISSUE_TEMPLATE/iceberg_bug_report.yml | 28 ++++++++
.github/ISSUE_TEMPLATE/iceberg_improvement.yml | 11 ++++
.github/ISSUE_TEMPLATE/iceberg_question.yml | 11 ++++
.github/dependabot.yml | 32 ++++++++++
.github/workflows/license_check.yml | 28 ++++++++
.github/workflows/python-ci-docs.yml | 56 ++++++++++++++++
.github/workflows/python-ci.yml | 53 ++++++++++++++++
.github/workflows/python-integration.yml | 46 ++++++++++++++
.github/workflows/python-release.yml | 88 ++++++++++++++++++++++++++
.github/workflows/stale.yml | 49 ++++++++++++++
.gitignore | 52 +++++++++++++++
.pre-commit-config.yaml | 19 ++----
dev/.rat-excludes | 2 +
13 files changed, 461 insertions(+), 14 deletions(-)
diff --git a/.github/ISSUE_TEMPLATE/iceberg_bug_report.yml b/.github/ISSUE_TEMPLATE/iceberg_bug_report.yml
new file mode 100644
index 0000000..41110cf
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/iceberg_bug_report.yml
@@ -0,0 +1,28 @@
+---
+name: Iceberg Bug report 🐞
+description: Problems, bugs and issues with Apache Iceberg
+labels: ["kind:bug"]
+body:
+ - type: dropdown
+ attributes:
+ label: Apache Iceberg version
+ description: What Apache Iceberg version are you using?
+ multiple: false
+ options:
+ - "0.5.0 (latest release)"
+ - "0.4.0"
+ - "0.3.0"
+ - "0.2.0"
+ - "0.1.0"
+ - "main (development)"
+ validations:
+ required: false
+ - type: textarea
+ attributes:
+ label: Please describe the bug 🐞
+ description: >
+ Please describe the problem, what to expect, and how to reproduce.
+ Feel free to include stacktraces and the Iceberg catalog configuration.
+ You can include files by dragging and dropping them here.
+ validations:
+ required: true
diff --git a/.github/ISSUE_TEMPLATE/iceberg_improvement.yml b/.github/ISSUE_TEMPLATE/iceberg_improvement.yml
new file mode 100644
index 0000000..0429236
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/iceberg_improvement.yml
@@ -0,0 +1,11 @@
+---
+name: Iceberg Improvement / Feature Request
+description: New features with Apache Iceberg
+labels: ["kind:feature request"]
+body:
+ - type: textarea
+ attributes:
+ label: Feature Request / Improvement
+ description: Please describe the feature and elaborate on the use case and motivation behind it
+ validations:
+ required: true
diff --git a/.github/ISSUE_TEMPLATE/iceberg_question.yml b/.github/ISSUE_TEMPLATE/iceberg_question.yml
new file mode 100644
index 0000000..cf9997f
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/iceberg_question.yml
@@ -0,0 +1,11 @@
+---
+name: Iceberg Question
+description: Questions around Apache Iceberg
+labels: ["kind:question"]
+body:
+ - type: textarea
+ attributes:
+ label: Question
+ description: What is your question?
+ validations:
+ required: true
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
new file mode 100644
index 0000000..f851b8a
--- /dev/null
+++ b/.github/dependabot.yml
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+version: 2
+updates:
+ - package-ecosystem: "pip"
+ directory: "/"
+ schedule:
+ interval: "weekly"
+ day: "sunday"
+ open-pull-requests-limit: 5
+ - package-ecosystem: "github-actions"
+ directory: "/"
+ schedule:
+ interval: "weekly"
+ day: "sunday"
diff --git a/.github/workflows/license_check.yml b/.github/workflows/license_check.yml
new file mode 100644
index 0000000..9408a3a
--- /dev/null
+++ b/.github/workflows/license_check.yml
@@ -0,0 +1,28 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Run License Check"
+on: pull_request
+
+jobs:
+ rat:
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - run: dev/check-license
diff --git a/.github/workflows/python-ci-docs.yml b/.github/workflows/python-ci-docs.yml
new file mode 100644
index 0000000..fc65fc9
--- /dev/null
+++ b/.github/workflows/python-ci-docs.yml
@@ -0,0 +1,56 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Python Docs"
+on:
+ workflow_dispatch:
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+ docs:
+ runs-on: ubuntu-22.04
+
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-python@v4
+ with:
+ python-version: ${{ matrix.python }}
+ - name: Install
+ working-directory: ./mkdocs
+ run: pip install -r requirements.txt
+ - name: Build
+ working-directory: ./mkdocs
+ run: mkdocs build --strict
+ - name: Copy
+ working-directory: ./mkdocs
+ run: mv ./site /tmp/site
+ - name: Push changes to gh-pages branch
+ run: |
+ git checkout --orphan gh-pages-tmp
+ git rm --quiet -rf .
+ cp -r /tmp/site/* .
+ git config --global user.name 'GitHub Actions'
+ git config --global user.email 'actions@github.com'
+ echo "py.iceberg.apache.org" > CNAME
+ git add --all
+ git commit -m 'Publish Python docs'
+ git push -f origin gh-pages-tmp:gh-pages || true
diff --git a/.github/workflows/python-ci.yml b/.github/workflows/python-ci.yml
new file mode 100644
index 0000000..ce08adc
--- /dev/null
+++ b/.github/workflows/python-ci.yml
@@ -0,0 +1,53 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Python CI"
+
+on:
+ push:
+ branches:
+ - 'main'
+ pull_request:
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+ lint-and-test:
+ runs-on: ubuntu-22.04
+ strategy:
+ matrix:
+ python: ['3.8', '3.9', '3.10', '3.11']
+
+ steps:
+ - uses: actions/checkout@v4
+ - name: Install poetry
+ run: make install-poetry
+ - uses: actions/setup-python@v4
+ with:
+ python-version: ${{ matrix.python }}
+ cache: poetry
+ cache-dependency-path: ./poetry.lock
+ - name: Install
+ run: make install-dependencies
+ - name: Linters
+ run: make lint
+ - name: Tests
+ run: make test-coverage
diff --git a/.github/workflows/python-integration.yml b/.github/workflows/python-integration.yml
new file mode 100644
index 0000000..b361f8d
--- /dev/null
+++ b/.github/workflows/python-integration.yml
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Python Integration"
+
+on:
+ push:
+ branches:
+ - 'main'
+ pull_request:
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+ integration-test:
+ runs-on: ubuntu-20.04
+
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 2
+ - name: Install
+ run: make install
+ - name: Run integration tests
+ run: make test-integration
+ - name: Show debug logs
+ if: ${{ failure() }}
+ run: docker-compose -f dev/docker-compose.yml logs
diff --git a/.github/workflows/python-release.yml b/.github/workflows/python-release.yml
new file mode 100644
index 0000000..6902775
--- /dev/null
+++ b/.github/workflows/python-release.yml
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Python Release"
+
+on:
+ workflow_dispatch:
+ inputs:
+ version:
+ description: 'Version'
+ type: string
+ default: 'master'
+
+
+jobs:
+ build_wheels:
+ name: Build wheels on ${{ matrix.os }}
+ runs-on: ${{ matrix.os }}
+ strategy:
+ matrix:
+ os: [ ubuntu-22.04, windows-2022, macos-11 ]
+
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+
+ - uses: actions/setup-python@v4
+ with:
+ python-version: '3.8'
+
+ - name: Install poetry
+ run: pip install poetry
+
+ - name: Set version
+ run: python -m poetry version "${{ inputs.version }}"
+ working-directory: ./python
+ if: "${{ github.event.inputs.version != 'master' }}"
+
+ # Publish the source distribution with the version that's in
+ # the repository, otherwise the tests will fail
+ - name: Compile source distribution
+ run: python3 -m poetry build --format=sdist
+ if: startsWith(matrix.os, 'ubuntu')
+ working-directory: ./python
+
+ - name: Build wheels
+ uses: pypa/cibuildwheel@v2.16.0
+ with:
+ output-dir: wheelhouse
+ config-file: "pyproject.toml"
+ env:
+ # Ignore 32 bit architectures
+ CIBW_ARCHS: "auto64"
+ CIBW_PROJECT_REQUIRES_PYTHON: ">=3.8,<3.12"
+ CIBW_TEST_REQUIRES: "pytest==7.4.2 moto==4.2.2"
+ CIBW_TEST_EXTRAS: "s3fs,glue"
+ CIBW_TEST_COMMAND: "pytest -Werror {project}/tests/avro/test_decoder.py"
+ # There is an upstream issue with installing on MacOSX
+ # https://github.com/pypa/cibuildwheel/issues/1603
+ # Ignore tests for pypy since not all dependencies are compiled for it
+ # and would require a local rust build chain
+ CIBW_TEST_SKIP: "pp* *macosx*"
+
+ - name: Add source distribution
+ if: startsWith(matrix.os, 'ubuntu')
+ run: ls -lah dist/* && cp dist/* wheelhouse/
+
+ - uses: actions/upload-artifact@v3
+ with:
+ name: "release-${{ github.event.inputs.version }}"
+ path: ./wheelhouse/*
diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml
new file mode 100644
index 0000000..566ae24
--- /dev/null
+++ b/.github/workflows/stale.yml
@@ -0,0 +1,49 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Close Stale Issues"
+on:
+ schedule:
+ - cron: '0 0 * * *'
+
+permissions:
+ # All other permissions are set to none
+ issues: write
+
+jobs:
+ stale:
+ if: github.repository_owner == 'apache'
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/stale@v8.0.0
+ with:
+ stale-issue-label: 'stale'
+ exempt-issue-labels: 'not-stale'
+ days-before-issue-stale: 180
+ days-before-issue-close: 14
+ # Only close stale issues, leave PRs alone
+ days-before-pr-stale: -1
+ stale-issue-message: >
+ This issue has been automatically marked as stale because it has been open for 180 days
+ with no activity. It will be closed in next 14 days if no further activity occurs. To
+ permanently prevent this issue from being considered stale, add the label 'not-stale',
+ but commenting on the issue is preferred when possible.
+ close-issue-message: >
+ This issue has been closed because it has not received any activity in the last 14 days
+ since being marked as 'stale'
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..7043f0e
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,52 @@
+*.swp
+.DS_Store
+.cache
+tmp/
+site
+
+# intellij files
+.idea
+.idea_modules/
+*.ipr
+*.iws
+*.iml
+out
+
+# rat library install location
+lib/
+
+__pycache__/
+*.py[cod]
+.eggs/
+.tox/
+env/
+venv/
+*.egg-info/
+test-reports
+build/
+dist/
+sdist/
+.coverage
+coverage.xml
+.pytest_cache/
+
+# vscode/eclipse files
+.classpath
+.project
+.settings
+bin/
+
+# Hive/metastore files
+metastore_db/
+
+# Spark/metastore files
+spark-warehouse/
+derby.log
+
+# Python stuff
+.mypy_cache/
+htmlcov
+
+pyiceberg/avro/decoder_fast.c
+pyiceberg/avro/*.html
+pyiceberg/avro/*.so
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 49785f1..b8bc6e9 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -15,8 +15,7 @@
# specific language governing permissions and limitations
# under the License.
---
-files: ^python/
-exclude: ^python/vendor/
+exclude: ^vendor/
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
@@ -36,26 +35,18 @@ repos:
rev: v5.10.1
hooks:
- id: isort
- args: [--settings-path=python/pyproject.toml]
+ args: [--settings-path=pyproject.toml]
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.3.0
hooks:
- id: mypy
args:
- [--install-types, --non-interactive, --config=python/pyproject.toml]
- - repo: https://github.com/pre-commit/mirrors-prettier
- rev: v3.0.0-alpha.9-for-vscode
- hooks:
- - id: prettier
- args: [--plugin=prettier-plugin-toml]
- additional_dependencies:
- - prettier@2.7.1
- - prettier-plugin-toml@0.3.1
+ [--install-types, --non-interactive, --config=pyproject.toml]
- repo: https://github.com/hadialqattan/pycln
rev: v2.1.5
hooks:
- id: pycln
- args: [--config=python/pyproject.toml]
+ args: [--config=pyproject.toml]
- repo: https://github.com/asottile/pyupgrade
rev: v3.4.0
hooks:
@@ -65,7 +56,7 @@ repos:
rev: v3.0.0a6
hooks:
- id: pylint
- args: [--rcfile=python/pylintrc]
+ args: [--rcfile=pylintrc]
- repo: https://github.com/pycqa/flake8
rev: "6.0.0"
hooks:
diff --git a/dev/.rat-excludes b/dev/.rat-excludes
index fb90e7e..dd4a891 100644
--- a/dev/.rat-excludes
+++ b/dev/.rat-excludes
@@ -1,2 +1,4 @@
.rat-excludes
build
+.gitignore
+poetry.lock
[iceberg-python] 04/10: Exclude Pydantic 2.4.0, 2.4.1 (#38)
Posted by fo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fokko pushed a commit to branch pyiceberg-0.5.x
in repository https://gitbox.apache.org/repos/asf/iceberg-python.git
commit 9fd9998ddddbff54dfd978bee5de1b4a68ceb573
Author: Sung Yun <10...@users.noreply.github.com>
AuthorDate: Tue Oct 3 14:31:57 2023 -0400
Exclude Pydantic 2.4.0, 2.4.1 (#38)
Similar to https://github.com/apache/iceberg/pull/8647, I think we would need to exclude 2.4.0, 2.4.1 here on the overall project's dependencies.
Given that pydantic has not de-shelved these faulty versions, we might not need to exclude them from the PyIceberg dependencies in order to prevent the dependency resolver from from installing pydantic 2.4.0 or 2.4.1
When pip installing pyiceberg 0.5.0 without version constraints, we see the following error on load_catalog function call:
```
KeyError Traceback (most recent call last)
/tmp/ipykernel_252/2476882581.py in <cell line: 1>()
----> 1 from pyiceberg.catalog import load_catalog
2
3 catalog = load_catalog("lacus")
/lib/python3.10/site-packages/pyiceberg/catalog/__init__.py in <module>
40 from pyiceberg.partitioning import UNPARTITIONED_PARTITION_SPEC, PartitionSpec
41 from pyiceberg.schema import Schema
---> 42 from pyiceberg.serializers import ToOutputFile
43 from pyiceberg.table import (
44 CommitTableRequest,
/lib/python3.10/site-packages/pyiceberg/serializers.py in <module>
23
24 from pyiceberg.io import InputFile, InputStream, OutputFile
---> 25 from pyiceberg.table.metadata import TableMetadata, TableMetadataUtil
26
27 GZIP = "gzip"
/lib/python3.10/site-packages/pyiceberg/table/__init__.py in <module>
69 visit,
70 )
---> 71 from pyiceberg.table.metadata import INITIAL_SEQUENCE_NUMBER, TableMetadata
72 from pyiceberg.table.snapshots import Snapshot, SnapshotLogEntry
73 from pyiceberg.table.sorting import SortOrder
/lib/python3.10/site-packages/pyiceberg/table/metadata.py in <module>
344
345
--> 346 class TableMetadataV2(TableMetadataCommonFields, IcebergBaseModel):
347 """Represents version 2 of the Table Metadata.
348
/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py in __new__(mcs, cls_name, bases, namespace, __pydantic_generic_metadata__, __pydantic_reset_parent_namespace__, **kwargs)
182 types_namespace = get_cls_types_namespace(cls, parent_namespace)
183 set_model_fields(cls, bases, config_wrapper, types_namespace)
--> 184 complete_model_class(
185 cls,
186 cls_name,
/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py in complete_model_class(cls, cls_name, config_wrapper, raise_errors, types_namespace)
493 return False
494
--> 495 schema = apply_discriminators(simplify_schema_references(schema))
496
497 # debug(schema)
/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py in simplify_schema_references(schema)
517 return s
518
--> 519 schema = walk_core_schema(schema, count_refs)
520
521 assert all(c == 0 for c in state['current_recursion_ref_count'].values()), 'this is a bug! please report it'
/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py in walk_core_schema(schema, f)
437 core_schema.CoreSchema: A processed CoreSchema.
438 """
--> 439 return f(schema, _dispatch)
440
441
/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py in count_refs(s, recurse)
513
514 state['current_recursion_ref_count'][ref] += 1
--> 515 recurse(state['definitions'][ref], count_refs)
516 state['current_recursion_ref_count'][ref] -= 1
517 return s
/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py in walk(self, schema, f)
213
214 def walk(self, schema: core_schema.CoreSchema, f: Walk) -> core_schema.CoreSchema:
--> 215 return f(schema, self._walk)
216
217 def _walk(self, schema: core_schema.CoreSchema, f: Walk) -> core_schema.CoreSchema:
/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py in count_refs(s, recurse)
501
502 if s['type'] != 'definition-ref':
--> 503 return recurse(s, count_refs)
504 ref = s['schema_ref']
505 state['ref_counts'][ref] += 1
...
...
...
/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py in walk(self, schema, f)
213
214 def walk(self, schema: core_schema.CoreSchema, f: Walk) -> core_schema.CoreSchema:
--> 215 return f(schema, self._walk)
216
217 def _walk(self, schema: core_schema.CoreSchema, f: Walk) -> core_schema.CoreSchema:
/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py in count_refs(s, recurse)
513
514 state['current_recursion_ref_count'][ref] += 1
--> 515 recurse(state['definitions'][ref], count_refs)
516 state['current_recursion_ref_count'][ref] -= 1
517 return s
KeyError: 'pyiceberg.types.NestedField:43624544'
```
---
pyproject.toml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pyproject.toml b/pyproject.toml
index c3d61e2..1d15df1 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -52,7 +52,7 @@ requests = ">=2.20.0,<3.0.0"
click = ">=7.1.1,<9.0.0"
rich = ">=10.11.0,<14.0.0"
strictyaml = ">=1.7.0,<2.0.0" # CVE-2020-14343 was fixed in 5.4.
-pydantic = ">=2.0,<3.0"
+pydantic = ">=2.0,<3.0,!=2.4.0,!=2.4.1" # 2.4.0, 2.4.1 has a critical bug
sortedcontainers = "2.4.0"
fsspec = ">=2023.1.0,<2024.1.0"
pyparsing = ">=3.1.0,<4.0.0"
[iceberg-python] 10/10: Bump version to 0.5.1 (#68)
Posted by fo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fokko pushed a commit to branch pyiceberg-0.5.x
in repository https://gitbox.apache.org/repos/asf/iceberg-python.git
commit c42317f2b755f4c9fc115e483b6f894382b44bdd
Author: Fokko Driesprong <fo...@tabular.io>
AuthorDate: Sat Oct 14 12:19:06 2023 +0200
Bump version to 0.5.1 (#68)
---
pyiceberg/__init__.py | 2 +-
pyproject.toml | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/pyiceberg/__init__.py b/pyiceberg/__init__.py
index c95a617..8a72b22 100644
--- a/pyiceberg/__init__.py
+++ b/pyiceberg/__init__.py
@@ -15,4 +15,4 @@
# specific language governing permissions and limitations
# under the License.
-__version__ = "0.5.0"
+__version__ = "0.5.1"
diff --git a/pyproject.toml b/pyproject.toml
index 1d15df1..902513f 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -16,7 +16,7 @@
# under the License.
[tool.poetry]
name = "pyiceberg"
-version = "0.5.0"
+version = "0.5.1"
readme = "README.md"
homepage = "https://py.iceberg.apache.org/"
repository = "https://github.com/apache/iceberg/"
[iceberg-python] 03/10: docs: Update how to release (#34)
Posted by fo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fokko pushed a commit to branch pyiceberg-0.5.x
in repository https://gitbox.apache.org/repos/asf/iceberg-python.git
commit 6332f562a151338700012bca595adbcb9b9a4a6d
Author: Fokko Driesprong <fo...@tabular.io>
AuthorDate: Tue Oct 3 08:55:17 2023 +0200
docs: Update how to release (#34)
---
mkdocs/docs/how-to-release.md | 16 +++++++++++-----
mkdocs/docs/verify-release.md | 17 +++++++++++++----
2 files changed, 24 insertions(+), 9 deletions(-)
diff --git a/mkdocs/docs/how-to-release.md b/mkdocs/docs/how-to-release.md
index 37af8bb..32e8744 100644
--- a/mkdocs/docs/how-to-release.md
+++ b/mkdocs/docs/how-to-release.md
@@ -57,14 +57,16 @@ Before committing the files to the Apache SVN artifact distribution SVN hashes n
Go to [Github Actions and run the `Python release` action](https://github.com/apache/iceberg/actions/workflows/python-release.yml). **Set the version to master, since we cannot modify the source**. Download the zip, and sign the files:
```bash
-for name in $(ls release-master/pyiceberg-*.whl release-master/pyiceberg-*.tar.gz)
+cd release-master/
+
+for name in $(ls pyiceberg-*.whl pyiceberg-*.tar.gz)
do
gpg --yes --armor --local-user fokko@apache.org --output "${name}.asc" --detach-sig "${name}"
- shasum -a 512 "${name}.asc" > "${name}.asc.sha512"
+ shasum -a 512 "${name}" > "${name}.sha512"
done
```
-Now we can upload the files
+Now we can upload the files from the same directory:
```bash
export SVN_TMP_DIR=/tmp/iceberg-${VERSION_BRANCH}/
@@ -72,7 +74,7 @@ svn checkout https://dist.apache.org/repos/dist/dev/iceberg $SVN_TMP_DIR
export SVN_TMP_DIR_VERSIONED=${SVN_TMP_DIR}pyiceberg-$VERSION/
mkdir -p $SVN_TMP_DIR_VERSIONED
-cp release-master/* $SVN_TMP_DIR_VERSIONED
+cp * $SVN_TMP_DIR_VERSIONED
svn add $SVN_TMP_DIR_VERSIONED
svn ci -m "PyIceberg ${VERSION}" ${SVN_TMP_DIR_VERSIONED}
```
@@ -92,11 +94,15 @@ Final step is to generate the email to the dev mail list:
```bash
cat << EOF > release-announcement-email.txt
To: dev@iceberg.apache.org
-Subject: [VOTE] Release Apache PyIceberg $VERSION_WITHOUT_RC
+Subject: [VOTE] Release Apache PyIceberg $VERSION
Hi Everyone,
I propose that we release the following RC as the official PyIceberg $VERSION_WITHOUT_RC release.
+A summary of the high level features:
+
+* <Add summary by hand>
+
The commit ID is $LAST_COMMIT_ID
* This corresponds to the tag: $GIT_TAG ($GIT_TAG_HASH)
diff --git a/mkdocs/docs/verify-release.md b/mkdocs/docs/verify-release.md
index 5750d28..e57a05d 100644
--- a/mkdocs/docs/verify-release.md
+++ b/mkdocs/docs/verify-release.md
@@ -47,20 +47,29 @@ gpg --import KEYS
Next, verify the `.asc` file.
```sh
-gpg --verify pyiceberg-0.4.0-py3-none-any.whl.asc pyiceberg-0.4.0-py3-none-any.whl
+svn checkout https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-0.5.0rc1/ /tmp/pyiceberg/
+
+for name in $(ls /tmp/pyiceberg/pyiceberg-*.whl /tmp/pyiceberg/pyiceberg-*.tar.gz)
+do
+ gpg --verify ${name}.asc ${name}
+done
```
## Verifying checksums
```sh
-shasum -a 512 --check pyiceberg-0.4.0-py3-none-any.whl.sha512
+cd /tmp/pyiceberg/
+for name in $(ls /tmp/pyiceberg/pyiceberg-*.whl.asc.sha512 /tmp/pyiceberg/pyiceberg-*.tar.gz.asc.sha512)
+do
+ shasum -a 512 --check ${name}
+done
```
## Verifying License Documentation
```sh
-tar xzf pyiceberg-0.4.0.tar.gz
-cd pyiceberg-0.4.0
+tar xzf pyiceberg-0.5.0.tar.gz
+cd pyiceberg-0.5.0
```
Run RAT checks to validate license header:
[iceberg-python] 01/10: Python: Create HadoopFileSystem from netloc (merge request !1060) (#8596)
Posted by fo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fokko pushed a commit to branch pyiceberg-0.5.x
in repository https://gitbox.apache.org/repos/asf/iceberg-python.git
commit 6c8ea0effac0942ad4e880e5eef627473a354040
Author: frankliee <fr...@tencent.com>
AuthorDate: Fri Sep 22 19:25:38 2023 +0800
Python: Create HadoopFileSystem from netloc (merge request !1060) (#8596)
---
pyiceberg/io/pyarrow.py | 36 +++++++++++++++++++-----------------
tests/io/test_pyarrow.py | 27 +++++++++++++--------------
2 files changed, 32 insertions(+), 31 deletions(-)
diff --git a/pyiceberg/io/pyarrow.py b/pyiceberg/io/pyarrow.py
index c3ac86d..fa40d0b 100644
--- a/pyiceberg/io/pyarrow.py
+++ b/pyiceberg/io/pyarrow.py
@@ -284,24 +284,24 @@ class PyArrowFile(InputFile, OutputFile):
class PyArrowFileIO(FileIO):
- fs_by_scheme: Callable[[str], FileSystem]
+ fs_by_scheme: Callable[[str, Optional[str]], FileSystem]
def __init__(self, properties: Properties = EMPTY_DICT):
- self.fs_by_scheme: Callable[[str], FileSystem] = lru_cache(self._initialize_fs)
+ self.fs_by_scheme: Callable[[str, Optional[str]], FileSystem] = lru_cache(self._initialize_fs)
super().__init__(properties=properties)
@staticmethod
- def parse_location(location: str) -> Tuple[str, str]:
+ def parse_location(location: str) -> Tuple[str, str, str]:
"""Return the path without the scheme."""
uri = urlparse(location)
if not uri.scheme:
- return "file", os.path.abspath(location)
+ return "file", uri.netloc, os.path.abspath(location)
elif uri.scheme == "hdfs":
- return uri.scheme, location
+ return uri.scheme, uri.netloc, location
else:
- return uri.scheme, f"{uri.netloc}{uri.path}"
+ return uri.scheme, uri.netloc, f"{uri.netloc}{uri.path}"
- def _initialize_fs(self, scheme: str) -> FileSystem:
+ def _initialize_fs(self, scheme: str, netloc: Optional[str] = None) -> FileSystem:
if scheme in {"s3", "s3a", "s3n"}:
from pyarrow.fs import S3FileSystem
@@ -321,6 +321,8 @@ class PyArrowFileIO(FileIO):
from pyarrow.fs import HadoopFileSystem
hdfs_kwargs: Dict[str, Any] = {}
+ if netloc:
+ return HadoopFileSystem.from_uri(f"hdfs://{netloc}")
if host := self.properties.get(HDFS_HOST):
hdfs_kwargs["host"] = host
if port := self.properties.get(HDFS_PORT):
@@ -364,9 +366,9 @@ class PyArrowFileIO(FileIO):
Returns:
PyArrowFile: A PyArrowFile instance for the given location.
"""
- scheme, path = self.parse_location(location)
+ scheme, netloc, path = self.parse_location(location)
return PyArrowFile(
- fs=self.fs_by_scheme(scheme),
+ fs=self.fs_by_scheme(scheme, netloc),
location=location,
path=path,
buffer_size=int(self.properties.get(BUFFER_SIZE, ONE_MEGABYTE)),
@@ -381,9 +383,9 @@ class PyArrowFileIO(FileIO):
Returns:
PyArrowFile: A PyArrowFile instance for the given location.
"""
- scheme, path = self.parse_location(location)
+ scheme, netloc, path = self.parse_location(location)
return PyArrowFile(
- fs=self.fs_by_scheme(scheme),
+ fs=self.fs_by_scheme(scheme, netloc),
location=location,
path=path,
buffer_size=int(self.properties.get(BUFFER_SIZE, ONE_MEGABYTE)),
@@ -402,8 +404,8 @@ class PyArrowFileIO(FileIO):
an AWS error code 15.
"""
str_location = location.location if isinstance(location, (InputFile, OutputFile)) else location
- scheme, path = self.parse_location(str_location)
- fs = self.fs_by_scheme(scheme)
+ scheme, netloc, path = self.parse_location(str_location)
+ fs = self.fs_by_scheme(scheme, netloc)
try:
fs.delete_file(path)
@@ -575,7 +577,7 @@ def _get_file_format(file_format: FileFormat, **kwargs: Dict[str, Any]) -> ds.Fi
def _construct_fragment(fs: FileSystem, data_file: DataFile, file_format_kwargs: Dict[str, Any] = EMPTY_DICT) -> ds.Fragment:
- _, path = PyArrowFileIO.parse_location(data_file.file_path)
+ _, _, path = PyArrowFileIO.parse_location(data_file.file_path)
return _get_file_format(data_file.file_format, **file_format_kwargs).make_fragment(path, fs)
@@ -797,7 +799,7 @@ def _task_to_table(
if limit and sum(row_counts) >= limit:
return None
- _, path = PyArrowFileIO.parse_location(task.file.file_path)
+ _, _, path = PyArrowFileIO.parse_location(task.file.file_path)
arrow_format = ds.ParquetFileFormat(pre_buffer=True, buffer_size=(ONE_MEGABYTE * 8))
with fs.open_input_file(path) as fin:
fragment = arrow_format.make_fragment(fin)
@@ -906,9 +908,9 @@ def project_table(
Raises:
ResolveError: When an incompatible query is done.
"""
- scheme, _ = PyArrowFileIO.parse_location(table.location())
+ scheme, netloc, _ = PyArrowFileIO.parse_location(table.location())
if isinstance(table.io, PyArrowFileIO):
- fs = table.io.fs_by_scheme(scheme)
+ fs = table.io.fs_by_scheme(scheme, netloc)
else:
try:
from pyiceberg.io.fsspec import FsspecFileIO
diff --git a/tests/io/test_pyarrow.py b/tests/io/test_pyarrow.py
index 49e1c8b..8b62212 100644
--- a/tests/io/test_pyarrow.py
+++ b/tests/io/test_pyarrow.py
@@ -1529,17 +1529,16 @@ def test_writing_avro_file_gcs(generated_manifest_entry_file: str, pyarrow_filei
pyarrow_fileio_gcs.delete(f"gs://warehouse/{filename}")
-def test_parse_hdfs_location() -> None:
- locations = ["hdfs://127.0.0.1:9000/root/foo.txt", "hdfs://127.0.0.1/root/foo.txt"]
- for location in locations:
- schema, path = PyArrowFileIO.parse_location(location)
- assert schema == "hdfs"
- assert location == path
-
-
-def test_parse_local_location() -> None:
- locations = ["/root/foo.txt", "/root/tmp/foo.txt"]
- for location in locations:
- schema, path = PyArrowFileIO.parse_location(location)
- assert schema == "file"
- assert location == path
+def test_parse_location() -> None:
+ def check_results(location: str, expected_schema: str, expected_netloc: str, expected_uri: str) -> None:
+ schema, netloc, uri = PyArrowFileIO.parse_location(location)
+ assert schema == expected_schema
+ assert netloc == expected_netloc
+ assert uri == expected_uri
+
+ check_results("hdfs://127.0.0.1:9000/root/foo.txt", "hdfs", "127.0.0.1:9000", "hdfs://127.0.0.1:9000/root/foo.txt")
+ check_results("hdfs://127.0.0.1/root/foo.txt", "hdfs", "127.0.0.1", "hdfs://127.0.0.1/root/foo.txt")
+ check_results("hdfs://clusterA/root/foo.txt", "hdfs", "clusterA", "hdfs://clusterA/root/foo.txt")
+
+ check_results("/root/foo.txt", "file", "", "/root/foo.txt")
+ check_results("/root/tmp/foo.txt", "file", "", "/root/tmp/foo.txt")
[iceberg-python] 08/10: Expression: Fix multiple and/or expressions (#65)
Posted by fo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fokko pushed a commit to branch pyiceberg-0.5.x
in repository https://gitbox.apache.org/repos/asf/iceberg-python.git
commit 7e0041fb25493970a314f49988bd4fde5b2a8fb0
Author: Amogh Jahagirdar <am...@tabular.io>
AuthorDate: Fri Oct 13 11:33:13 2023 -0700
Expression: Fix multiple and/or expressions (#65)
---
pyiceberg/expressions/parser.py | 4 ++--
tests/expressions/test_parser.py | 8 ++++++++
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/pyiceberg/expressions/parser.py b/pyiceberg/expressions/parser.py
index d6d5bdb..a5452f5 100644
--- a/pyiceberg/expressions/parser.py
+++ b/pyiceberg/expressions/parser.py
@@ -233,11 +233,11 @@ def handle_not(result: ParseResults) -> Not:
def handle_and(result: ParseResults) -> And:
- return And(result[0][0], result[0][1])
+ return And(*result[0])
def handle_or(result: ParseResults) -> Or:
- return Or(result[0][0], result[0][1])
+ return Or(*result[0])
boolean_expression = infix_notation(
diff --git a/tests/expressions/test_parser.py b/tests/expressions/test_parser.py
index f4bebca..3b21835 100644
--- a/tests/expressions/test_parser.py
+++ b/tests/expressions/test_parser.py
@@ -160,6 +160,14 @@ def test_and_or_with_parens() -> None:
)
+def test_multiple_and_or() -> None:
+ assert And(EqualTo("foo", 1), EqualTo("bar", 2), EqualTo("baz", 3)) == parser.parse("foo = 1 and bar = 2 and baz = 3")
+ assert Or(EqualTo("foo", 1), EqualTo("foo", 2), EqualTo("foo", 3)) == parser.parse("foo = 1 or foo = 2 or foo = 3")
+ assert Or(
+ And(NotNull("foo"), LessThan("foo", 5)), And(GreaterThan("foo", 10), LessThan("foo", 100), IsNull("bar"))
+ ) == parser.parse("foo is not null and foo < 5 or (foo > 10 and foo < 100 and bar is null)")
+
+
def test_starts_with() -> None:
assert StartsWith("foo", "data") == parser.parse("foo LIKE 'data'")