You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "lidavidm (via GitHub)" <gi...@apache.org> on 2023/11/30 17:19:58 UTC

[I] python/adbc_driver_postgresql: integration tests are failing [arrow-adbc]

lidavidm opened a new issue, #1333:
URL: https://github.com/apache/arrow-adbc/issues/1333

   ```
   =================================== FAILURES ===================================
   _______________________ test_polars_write_database[ints] _______________________
   
   postgres_uri = 'postgresql://localhost:5432/postgres?user=postgres&***'
   df = shape: (4, 1)
   ┌──────┐
   │ ints │
   │ ---  │
   │ i64  │
   ╞══════╡
   │ 1    │
   │ 2    │
   │ 4    │
   │ 8    │
   └──────┘
   
       @pytest.mark.parametrize(
           "df",
           [
               "ints",
               "floats",
           ],
           indirect=True,
       )
       def test_polars_write_database(postgres_uri: str, df: "polars.DataFrame") -> None:
           table_name = f"polars_test_ingest_{uuid.uuid4().hex}"
           try:
   >           df.write_database(
                   table_name=table_name,
                   connection=postgres_uri,
                   # TODO(apache/arrow-adbc#541): polars doesn't map the semantics
                   # properly here, and one of their modes isn't supported
                   if_exists="replace",
                   engine="adbc",
               )
   
   python/adbc_driver_postgresql/tests/test_polars.py:71: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   /usr/share/miniconda3/envs/test/lib/python3.11/site-packages/polars/utils/deprecation.py:100: in wrapper
       return function(*args, **kwargs)
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   
   self = shape: (4, 1)
   ┌──────┐
   │ ints │
   │ ---  │
   │ i64  │
   ╞══════╡
   │ 1    │
   │ 2    │
   │ 4    │
   │ 8    │
   └──────┘
   table_name = 'polars_test_ingest_23929440c8084693b505cf596fe45b46'
   connection = 'postgresql://localhost:5432/postgres?user=postgres&***'
   
       @deprecate_renamed_parameter("connection_uri", "connection", version="0.18.9")
       def write_database(
           self,
           table_name: str,
           connection: str,
           *,
           if_exists: DbWriteMode = "fail",
           engine: DbWriteEngine = "sqlalchemy",
       ) -> None:
           """
           Write a polars frame to a database.
       
           Parameters
           ----------
           table_name
               Schema-qualified name of the table to create or append to in the target
               SQL database. If your table name contains special characters, it should
               be quoted.
           connection
               Connection URI string, for example:
       
               * "***server:port/database"
               * "sqlite:////path/to/database.db"
           if_exists : {'append', 'replace', 'fail'}
               The insert mode:
       
               * 'replace' will create a new database table, overwriting an existing one.
               * 'append' will append to an existing table.
               * 'fail' will fail if table already exists.
           engine : {'sqlalchemy', 'adbc'}
               Select the engine used for writing the data.
           """
           from polars.io.database import _open_adbc_connection
       
           def unpack_table_name(name: str) -> tuple[str | None, str]:
               """Unpack optionally qualified table name into schema/table pair."""
               from csv import reader as delimited_read
       
               table_ident = next(delimited_read([name], delimiter="."))
               if len(table_ident) > 2:
                   raise ValueError(f"`table_name` appears to be invalid: {name!r}")
               elif len(table_ident) > 1:
                   schema = table_ident[0]
                   tbl = table_ident[1]
               else:
                   schema = None
                   tbl = table_ident[0]
               return schema, tbl
       
           if engine == "adbc":
               try:
                   import adbc_driver_manager
       
                   adbc_version = parse_version(
                       getattr(adbc_driver_manager, "__version__", "0.0")
                   )
               except ModuleNotFoundError as exc:
                   raise ModuleNotFoundError(
                       "adbc_driver_manager not found"
                       "\n\nInstall Polars with: pip install adbc_driver_manager"
                   ) from exc
       
               if if_exists == "fail":
                   # if the table exists, 'create' will raise an error,
                   # resulting in behaviour equivalent to 'fail'
                   mode = "create"
               elif if_exists == "replace":
                   if adbc_version < (0, 7):
                       adbc_str_version = ".".join(str(v) for v in adbc_version)
   >                   raise ModuleNotFoundError(
                           f"`if_exists = 'replace'` requires ADBC version >= 0.7, found {adbc_str_version}"
                       )
   E                   ModuleNotFoundError: `if_exists = 'replace'` requires ADBC version >= 0.7, found 0.0.5174
   
   /usr/share/miniconda3/envs/test/lib/python3.11/site-packages/polars/dataframe/frame.py:3491: ModuleNotFoundError
   
   During handling of the above exception, another exception occurred:
   
   postgres_uri = 'postgresql://localhost:5432/postgres?user=postgres&***'
   df = shape: (4, 1)
   ┌──────┐
   │ ints │
   │ ---  │
   │ i64  │
   ╞══════╡
   │ 1    │
   │ 2    │
   │ 4    │
   │ 8    │
   └──────┘
   
       @pytest.mark.parametrize(
           "df",
           [
               "ints",
               "floats",
           ],
           indirect=True,
       )
       def test_polars_write_database(postgres_uri: str, df: "polars.DataFrame") -> None:
           table_name = f"polars_test_ingest_{uuid.uuid4().hex}"
           try:
               df.write_database(
                   table_name=table_name,
                   connection=postgres_uri,
                   # TODO(apache/arrow-adbc#541): polars doesn't map the semantics
                   # properly here, and one of their modes isn't supported
                   if_exists="replace",
                   engine="adbc",
               )
           finally:
               with dbapi.connect(postgres_uri) as conn:
                   with conn.cursor() as cursor:
   >                   cursor.execute(f"DROP TABLE {table_name}")
   
   python/adbc_driver_postgresql/tests/test_polars.py:82: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   python/adbc_driver_manager/adbc_driver_manager/dbapi.py:669: in execute
       handle, self._rowcount = self._stmt.execute_query()
   adbc_driver_manager/_lib.pyx:1106: in adbc_driver_manager._lib.AdbcStatement.execute_query
       ???
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   
   >   ???
   E   adbc_driver_manager.ProgrammingError: NOT_FOUND: [libpq] Failed to execute query: ERROR:  table "polars_test_ingest_23929440c8084693b505cf596fe45b46" does not exist
   E   
   E   Query was:DROP TABLE polars_test_ingest_23929440c8084693b505cf596fe45b46. SQLSTATE: 42P01
   
   adbc_driver_manager/_lib.pyx:227: ProgrammingError
   ______________________ test_polars_write_database[floats] ______________________
   
   postgres_uri = 'postgresql://localhost:5432/postgres?user=postgres&***'
   df = shape: (4, 1)
   ┌────────┐
   │ floats │
   │ ---    │
   │ f64    │
   ╞════════╡
   │ 1.0    │
   │ 2.0    │
   │ 4.0    │
   │ 8.0    │
   └────────┘
   
       @pytest.mark.parametrize(
           "df",
           [
               "ints",
               "floats",
           ],
           indirect=True,
       )
       def test_polars_write_database(postgres_uri: str, df: "polars.DataFrame") -> None:
           table_name = f"polars_test_ingest_{uuid.uuid4().hex}"
           try:
   >           df.write_database(
                   table_name=table_name,
                   connection=postgres_uri,
                   # TODO(apache/arrow-adbc#541): polars doesn't map the semantics
                   # properly here, and one of their modes isn't supported
                   if_exists="replace",
                   engine="adbc",
               )
   
   python/adbc_driver_postgresql/tests/test_polars.py:71: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   /usr/share/miniconda3/envs/test/lib/python3.11/site-packages/polars/utils/deprecation.py:100: in wrapper
       return function(*args, **kwargs)
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   
   self = shape: (4, 1)
   ┌────────┐
   │ floats │
   │ ---    │
   │ f64    │
   ╞════════╡
   │ 1.0    │
   │ 2.0    │
   │ 4.0    │
   │ 8.0    │
   └────────┘
   table_name = 'polars_test_ingest_7dc879158fdf47f5be7712cc17531132'
   connection = 'postgresql://localhost:5432/postgres?user=postgres&***'
   
       @deprecate_renamed_parameter("connection_uri", "connection", version="0.18.9")
       def write_database(
           self,
           table_name: str,
           connection: str,
           *,
           if_exists: DbWriteMode = "fail",
           engine: DbWriteEngine = "sqlalchemy",
       ) -> None:
           """
           Write a polars frame to a database.
       
           Parameters
           ----------
           table_name
               Schema-qualified name of the table to create or append to in the target
               SQL database. If your table name contains special characters, it should
               be quoted.
           connection
               Connection URI string, for example:
       
               * "***server:port/database"
               * "sqlite:////path/to/database.db"
           if_exists : {'append', 'replace', 'fail'}
               The insert mode:
       
               * 'replace' will create a new database table, overwriting an existing one.
               * 'append' will append to an existing table.
               * 'fail' will fail if table already exists.
           engine : {'sqlalchemy', 'adbc'}
               Select the engine used for writing the data.
           """
           from polars.io.database import _open_adbc_connection
       
           def unpack_table_name(name: str) -> tuple[str | None, str]:
               """Unpack optionally qualified table name into schema/table pair."""
               from csv import reader as delimited_read
       
               table_ident = next(delimited_read([name], delimiter="."))
               if len(table_ident) > 2:
                   raise ValueError(f"`table_name` appears to be invalid: {name!r}")
               elif len(table_ident) > 1:
                   schema = table_ident[0]
                   tbl = table_ident[1]
               else:
                   schema = None
                   tbl = table_ident[0]
               return schema, tbl
       
           if engine == "adbc":
               try:
                   import adbc_driver_manager
       
                   adbc_version = parse_version(
                       getattr(adbc_driver_manager, "__version__", "0.0")
                   )
               except ModuleNotFoundError as exc:
                   raise ModuleNotFoundError(
                       "adbc_driver_manager not found"
                       "\n\nInstall Polars with: pip install adbc_driver_manager"
                   ) from exc
       
               if if_exists == "fail":
                   # if the table exists, 'create' will raise an error,
                   # resulting in behaviour equivalent to 'fail'
                   mode = "create"
               elif if_exists == "replace":
                   if adbc_version < (0, 7):
                       adbc_str_version = ".".join(str(v) for v in adbc_version)
   >                   raise ModuleNotFoundError(
                           f"`if_exists = 'replace'` requires ADBC version >= 0.7, found {adbc_str_version}"
                       )
   E                   ModuleNotFoundError: `if_exists = 'replace'` requires ADBC version >= 0.7, found 0.0.5174
   
   /usr/share/miniconda3/envs/test/lib/python3.11/site-packages/polars/dataframe/frame.py:3491: ModuleNotFoundError
   
   During handling of the above exception, another exception occurred:
   
   postgres_uri = 'postgresql://localhost:5432/postgres?user=postgres&***'
   df = shape: (4, 1)
   ┌────────┐
   │ floats │
   │ ---    │
   │ f64    │
   ╞════════╡
   │ 1.0    │
   │ 2.0    │
   │ 4.0    │
   │ 8.0    │
   └────────┘
   
       @pytest.mark.parametrize(
           "df",
           [
               "ints",
               "floats",
           ],
           indirect=True,
       )
       def test_polars_write_database(postgres_uri: str, df: "polars.DataFrame") -> None:
           table_name = f"polars_test_ingest_{uuid.uuid4().hex}"
           try:
               df.write_database(
                   table_name=table_name,
                   connection=postgres_uri,
                   # TODO(apache/arrow-adbc#541): polars doesn't map the semantics
                   # properly here, and one of their modes isn't supported
                   if_exists="replace",
                   engine="adbc",
               )
           finally:
               with dbapi.connect(postgres_uri) as conn:
                   with conn.cursor() as cursor:
   >                   cursor.execute(f"DROP TABLE {table_name}")
   
   python/adbc_driver_postgresql/tests/test_polars.py:82: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   python/adbc_driver_manager/adbc_driver_manager/dbapi.py:669: in execute
       handle, self._rowcount = self._stmt.execute_query()
   adbc_driver_manager/_lib.pyx:1106: in adbc_driver_manager._lib.AdbcStatement.execute_query
       ???
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   
   >   ???
   E   adbc_driver_manager.ProgrammingError: NOT_FOUND: [libpq] Failed to execute query: ERROR:  table "polars_test_ingest_7dc879158fdf47f5be7712cc17531132" does not exist
   E   
   E   Query was:DROP TABLE polars_test_ingest_7dc879158fdf47f5be7712cc17531132. SQLSTATE: 42P01
   
   adbc_driver_manager/_lib.pyx:227: ProgrammingError
   =========================== short test summary info ============================
   FAILED python/adbc_driver_postgresql/tests/test_polars.py::test_polars_write_database[ints] - adbc_driver_manager.ProgrammingError: NOT_FOUND: [libpq] Failed to execute query: ERROR:  table "polars_test_ingest_23929440c8084693b505cf596fe45b46" does not exist
   
   Query was:DROP TABLE polars_test_ingest_23929440c8084693b505cf596fe45b46. SQLSTATE: 42P01
   FAILED python/adbc_driver_postgresql/tests/test_polars.py::test_polars_write_database[floats] - adbc_driver_manager.ProgrammingError: NOT_FOUND: [libpq] Failed to execute query: ERROR:  table "polars_test_ingest_7dc879158fdf47f5be7712cc17531132" does not exist
   
   Query was:DROP TABLE polars_test_ingest_7dc879158fdf47f5be7712cc17531132. SQLSTATE: 42P01
   ========================= 2 failed, 20 passed in 3.40s =========================
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] python/adbc_driver_postgresql: integration tests are failing [arrow-adbc]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #1333:
URL: https://github.com/apache/arrow-adbc/issues/1333#issuecomment-1874413676

   Closing in favor of #1363


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] python/adbc_driver_postgresql: integration tests are failing [arrow-adbc]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #1333:
URL: https://github.com/apache/arrow-adbc/issues/1333#issuecomment-1836083476

   Ah...That is how the release process works (using maintenance branches), which is a holdover from the main Arrow project (which creates a branch and cherry-picks commits onto it as needed). So either we can adjust CI to force/fake a particular version, or we could consider just releasing directly off of main.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] python/adbc_driver_postgresql: integration tests are failing [arrow-adbc]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #1333:
URL: https://github.com/apache/arrow-adbc/issues/1333#issuecomment-1843382954

   So continuing from #1337 we should set SETUPTOOLS_SCM_PRETEND_VERSION in our various pipelines and have the bump-version release script update this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] python/adbc_driver_postgresql: integration tests are failing [arrow-adbc]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm closed issue #1333: python/adbc_driver_postgresql: integration tests are failing
URL: https://github.com/apache/arrow-adbc/issues/1333


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] python/adbc_driver_postgresql: integration tests are failing [arrow-adbc]

Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on issue #1333:
URL: https://github.com/apache/arrow-adbc/issues/1333#issuecomment-1835551181

   I think this started with this upstream PR in polars:
   
   https://github.com/pola-rs/polars/pull/12713
   
   It looks like in arrow-adbc we are driving the release number off of what `git describe --long --always` produces:
   
   https://github.com/apache/arrow-adbc/blob/163bc03664fc6c0a855364b96dd9374b097373cb/python/adbc_driver_postgresql/adbc_driver_postgresql/_version.py#L117
   
   But it looks like the tags that exist don't exist on the main branch, and the commits they reference are on the maintenance branches (were these cherry-picked?) so when building from source you always end up with version `0.0.0+dirty`
   
   Is it intentional for tags not to exist on the main branch or a msitake?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org