You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "lidavidm (via GitHub)" <gi...@apache.org> on 2023/11/30 17:19:58 UTC
[I] python/adbc_driver_postgresql: integration tests are failing [arrow-adbc]
lidavidm opened a new issue, #1333:
URL: https://github.com/apache/arrow-adbc/issues/1333
```
=================================== FAILURES ===================================
_______________________ test_polars_write_database[ints] _______________________
postgres_uri = 'postgresql://localhost:5432/postgres?user=postgres&***'
df = shape: (4, 1)
┌──────┐
│ ints │
│ --- │
│ i64 │
╞══════╡
│ 1 │
│ 2 │
│ 4 │
│ 8 │
└──────┘
@pytest.mark.parametrize(
"df",
[
"ints",
"floats",
],
indirect=True,
)
def test_polars_write_database(postgres_uri: str, df: "polars.DataFrame") -> None:
table_name = f"polars_test_ingest_{uuid.uuid4().hex}"
try:
> df.write_database(
table_name=table_name,
connection=postgres_uri,
# TODO(apache/arrow-adbc#541): polars doesn't map the semantics
# properly here, and one of their modes isn't supported
if_exists="replace",
engine="adbc",
)
python/adbc_driver_postgresql/tests/test_polars.py:71:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/share/miniconda3/envs/test/lib/python3.11/site-packages/polars/utils/deprecation.py:100: in wrapper
return function(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = shape: (4, 1)
┌──────┐
│ ints │
│ --- │
│ i64 │
╞══════╡
│ 1 │
│ 2 │
│ 4 │
│ 8 │
└──────┘
table_name = 'polars_test_ingest_23929440c8084693b505cf596fe45b46'
connection = 'postgresql://localhost:5432/postgres?user=postgres&***'
@deprecate_renamed_parameter("connection_uri", "connection", version="0.18.9")
def write_database(
self,
table_name: str,
connection: str,
*,
if_exists: DbWriteMode = "fail",
engine: DbWriteEngine = "sqlalchemy",
) -> None:
"""
Write a polars frame to a database.
Parameters
----------
table_name
Schema-qualified name of the table to create or append to in the target
SQL database. If your table name contains special characters, it should
be quoted.
connection
Connection URI string, for example:
* "***server:port/database"
* "sqlite:////path/to/database.db"
if_exists : {'append', 'replace', 'fail'}
The insert mode:
* 'replace' will create a new database table, overwriting an existing one.
* 'append' will append to an existing table.
* 'fail' will fail if table already exists.
engine : {'sqlalchemy', 'adbc'}
Select the engine used for writing the data.
"""
from polars.io.database import _open_adbc_connection
def unpack_table_name(name: str) -> tuple[str | None, str]:
"""Unpack optionally qualified table name into schema/table pair."""
from csv import reader as delimited_read
table_ident = next(delimited_read([name], delimiter="."))
if len(table_ident) > 2:
raise ValueError(f"`table_name` appears to be invalid: {name!r}")
elif len(table_ident) > 1:
schema = table_ident[0]
tbl = table_ident[1]
else:
schema = None
tbl = table_ident[0]
return schema, tbl
if engine == "adbc":
try:
import adbc_driver_manager
adbc_version = parse_version(
getattr(adbc_driver_manager, "__version__", "0.0")
)
except ModuleNotFoundError as exc:
raise ModuleNotFoundError(
"adbc_driver_manager not found"
"\n\nInstall Polars with: pip install adbc_driver_manager"
) from exc
if if_exists == "fail":
# if the table exists, 'create' will raise an error,
# resulting in behaviour equivalent to 'fail'
mode = "create"
elif if_exists == "replace":
if adbc_version < (0, 7):
adbc_str_version = ".".join(str(v) for v in adbc_version)
> raise ModuleNotFoundError(
f"`if_exists = 'replace'` requires ADBC version >= 0.7, found {adbc_str_version}"
)
E ModuleNotFoundError: `if_exists = 'replace'` requires ADBC version >= 0.7, found 0.0.5174
/usr/share/miniconda3/envs/test/lib/python3.11/site-packages/polars/dataframe/frame.py:3491: ModuleNotFoundError
During handling of the above exception, another exception occurred:
postgres_uri = 'postgresql://localhost:5432/postgres?user=postgres&***'
df = shape: (4, 1)
┌──────┐
│ ints │
│ --- │
│ i64 │
╞══════╡
│ 1 │
│ 2 │
│ 4 │
│ 8 │
└──────┘
@pytest.mark.parametrize(
"df",
[
"ints",
"floats",
],
indirect=True,
)
def test_polars_write_database(postgres_uri: str, df: "polars.DataFrame") -> None:
table_name = f"polars_test_ingest_{uuid.uuid4().hex}"
try:
df.write_database(
table_name=table_name,
connection=postgres_uri,
# TODO(apache/arrow-adbc#541): polars doesn't map the semantics
# properly here, and one of their modes isn't supported
if_exists="replace",
engine="adbc",
)
finally:
with dbapi.connect(postgres_uri) as conn:
with conn.cursor() as cursor:
> cursor.execute(f"DROP TABLE {table_name}")
python/adbc_driver_postgresql/tests/test_polars.py:82:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
python/adbc_driver_manager/adbc_driver_manager/dbapi.py:669: in execute
handle, self._rowcount = self._stmt.execute_query()
adbc_driver_manager/_lib.pyx:1106: in adbc_driver_manager._lib.AdbcStatement.execute_query
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E adbc_driver_manager.ProgrammingError: NOT_FOUND: [libpq] Failed to execute query: ERROR: table "polars_test_ingest_23929440c8084693b505cf596fe45b46" does not exist
E
E Query was:DROP TABLE polars_test_ingest_23929440c8084693b505cf596fe45b46. SQLSTATE: 42P01
adbc_driver_manager/_lib.pyx:227: ProgrammingError
______________________ test_polars_write_database[floats] ______________________
postgres_uri = 'postgresql://localhost:5432/postgres?user=postgres&***'
df = shape: (4, 1)
┌────────┐
│ floats │
│ --- │
│ f64 │
╞════════╡
│ 1.0 │
│ 2.0 │
│ 4.0 │
│ 8.0 │
└────────┘
@pytest.mark.parametrize(
"df",
[
"ints",
"floats",
],
indirect=True,
)
def test_polars_write_database(postgres_uri: str, df: "polars.DataFrame") -> None:
table_name = f"polars_test_ingest_{uuid.uuid4().hex}"
try:
> df.write_database(
table_name=table_name,
connection=postgres_uri,
# TODO(apache/arrow-adbc#541): polars doesn't map the semantics
# properly here, and one of their modes isn't supported
if_exists="replace",
engine="adbc",
)
python/adbc_driver_postgresql/tests/test_polars.py:71:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/share/miniconda3/envs/test/lib/python3.11/site-packages/polars/utils/deprecation.py:100: in wrapper
return function(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = shape: (4, 1)
┌────────┐
│ floats │
│ --- │
│ f64 │
╞════════╡
│ 1.0 │
│ 2.0 │
│ 4.0 │
│ 8.0 │
└────────┘
table_name = 'polars_test_ingest_7dc879158fdf47f5be7712cc17531132'
connection = 'postgresql://localhost:5432/postgres?user=postgres&***'
@deprecate_renamed_parameter("connection_uri", "connection", version="0.18.9")
def write_database(
self,
table_name: str,
connection: str,
*,
if_exists: DbWriteMode = "fail",
engine: DbWriteEngine = "sqlalchemy",
) -> None:
"""
Write a polars frame to a database.
Parameters
----------
table_name
Schema-qualified name of the table to create or append to in the target
SQL database. If your table name contains special characters, it should
be quoted.
connection
Connection URI string, for example:
* "***server:port/database"
* "sqlite:////path/to/database.db"
if_exists : {'append', 'replace', 'fail'}
The insert mode:
* 'replace' will create a new database table, overwriting an existing one.
* 'append' will append to an existing table.
* 'fail' will fail if table already exists.
engine : {'sqlalchemy', 'adbc'}
Select the engine used for writing the data.
"""
from polars.io.database import _open_adbc_connection
def unpack_table_name(name: str) -> tuple[str | None, str]:
"""Unpack optionally qualified table name into schema/table pair."""
from csv import reader as delimited_read
table_ident = next(delimited_read([name], delimiter="."))
if len(table_ident) > 2:
raise ValueError(f"`table_name` appears to be invalid: {name!r}")
elif len(table_ident) > 1:
schema = table_ident[0]
tbl = table_ident[1]
else:
schema = None
tbl = table_ident[0]
return schema, tbl
if engine == "adbc":
try:
import adbc_driver_manager
adbc_version = parse_version(
getattr(adbc_driver_manager, "__version__", "0.0")
)
except ModuleNotFoundError as exc:
raise ModuleNotFoundError(
"adbc_driver_manager not found"
"\n\nInstall Polars with: pip install adbc_driver_manager"
) from exc
if if_exists == "fail":
# if the table exists, 'create' will raise an error,
# resulting in behaviour equivalent to 'fail'
mode = "create"
elif if_exists == "replace":
if adbc_version < (0, 7):
adbc_str_version = ".".join(str(v) for v in adbc_version)
> raise ModuleNotFoundError(
f"`if_exists = 'replace'` requires ADBC version >= 0.7, found {adbc_str_version}"
)
E ModuleNotFoundError: `if_exists = 'replace'` requires ADBC version >= 0.7, found 0.0.5174
/usr/share/miniconda3/envs/test/lib/python3.11/site-packages/polars/dataframe/frame.py:3491: ModuleNotFoundError
During handling of the above exception, another exception occurred:
postgres_uri = 'postgresql://localhost:5432/postgres?user=postgres&***'
df = shape: (4, 1)
┌────────┐
│ floats │
│ --- │
│ f64 │
╞════════╡
│ 1.0 │
│ 2.0 │
│ 4.0 │
│ 8.0 │
└────────┘
@pytest.mark.parametrize(
"df",
[
"ints",
"floats",
],
indirect=True,
)
def test_polars_write_database(postgres_uri: str, df: "polars.DataFrame") -> None:
table_name = f"polars_test_ingest_{uuid.uuid4().hex}"
try:
df.write_database(
table_name=table_name,
connection=postgres_uri,
# TODO(apache/arrow-adbc#541): polars doesn't map the semantics
# properly here, and one of their modes isn't supported
if_exists="replace",
engine="adbc",
)
finally:
with dbapi.connect(postgres_uri) as conn:
with conn.cursor() as cursor:
> cursor.execute(f"DROP TABLE {table_name}")
python/adbc_driver_postgresql/tests/test_polars.py:82:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
python/adbc_driver_manager/adbc_driver_manager/dbapi.py:669: in execute
handle, self._rowcount = self._stmt.execute_query()
adbc_driver_manager/_lib.pyx:1106: in adbc_driver_manager._lib.AdbcStatement.execute_query
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E adbc_driver_manager.ProgrammingError: NOT_FOUND: [libpq] Failed to execute query: ERROR: table "polars_test_ingest_7dc879158fdf47f5be7712cc17531132" does not exist
E
E Query was:DROP TABLE polars_test_ingest_7dc879158fdf47f5be7712cc17531132. SQLSTATE: 42P01
adbc_driver_manager/_lib.pyx:227: ProgrammingError
=========================== short test summary info ============================
FAILED python/adbc_driver_postgresql/tests/test_polars.py::test_polars_write_database[ints] - adbc_driver_manager.ProgrammingError: NOT_FOUND: [libpq] Failed to execute query: ERROR: table "polars_test_ingest_23929440c8084693b505cf596fe45b46" does not exist
Query was:DROP TABLE polars_test_ingest_23929440c8084693b505cf596fe45b46. SQLSTATE: 42P01
FAILED python/adbc_driver_postgresql/tests/test_polars.py::test_polars_write_database[floats] - adbc_driver_manager.ProgrammingError: NOT_FOUND: [libpq] Failed to execute query: ERROR: table "polars_test_ingest_7dc879158fdf47f5be7712cc17531132" does not exist
Query was:DROP TABLE polars_test_ingest_7dc879158fdf47f5be7712cc17531132. SQLSTATE: 42P01
========================= 2 failed, 20 passed in 3.40s =========================
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] python/adbc_driver_postgresql: integration tests are failing [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #1333:
URL: https://github.com/apache/arrow-adbc/issues/1333#issuecomment-1874413676
Closing in favor of #1363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] python/adbc_driver_postgresql: integration tests are failing [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #1333:
URL: https://github.com/apache/arrow-adbc/issues/1333#issuecomment-1836083476
Ah...That is how the release process works (using maintenance branches), which is a holdover from the main Arrow project (which creates a branch and cherry-picks commits onto it as needed). So either we can adjust CI to force/fake a particular version, or we could consider just releasing directly off of main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] python/adbc_driver_postgresql: integration tests are failing [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #1333:
URL: https://github.com/apache/arrow-adbc/issues/1333#issuecomment-1843382954
So continuing from #1337 we should set SETUPTOOLS_SCM_PRETEND_VERSION in our various pipelines and have the bump-version release script update this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] python/adbc_driver_postgresql: integration tests are failing [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm closed issue #1333: python/adbc_driver_postgresql: integration tests are failing
URL: https://github.com/apache/arrow-adbc/issues/1333
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] python/adbc_driver_postgresql: integration tests are failing [arrow-adbc]
Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on issue #1333:
URL: https://github.com/apache/arrow-adbc/issues/1333#issuecomment-1835551181
I think this started with this upstream PR in polars:
https://github.com/pola-rs/polars/pull/12713
It looks like in arrow-adbc we are driving the release number off of what `git describe --long --always` produces:
https://github.com/apache/arrow-adbc/blob/163bc03664fc6c0a855364b96dd9374b097373cb/python/adbc_driver_postgresql/adbc_driver_postgresql/_version.py#L117
But it looks like the tags that exist don't exist on the main branch, and the commits they reference are on the maintenance branches (were these cherry-picked?) so when building from source you always end up with version `0.0.0+dirty`
Is it intentional for tags not to exist on the main branch or a msitake?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org