You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "Habeeb556 (via GitHub)" <gi...@apache.org> on 2024/07/04 20:38:53 UTC
[I] MSSQL Error: list index out of range [superset]
Habeeb556 opened a new issue, #29494:
URL: https://github.com/apache/superset/issues/29494
### Bug description
I received the following error message:
`list index out of range`
This occurred when upgrading from `version 2.1.3`, which supports Python 3.8, to Superset `version 4.0.2`, which supports Python 3.11, using the MSSQL engine with `pyodbc version 5.0.1`.
### How to reproduce the bug
I think the problem lies with the pyodbc version, as it works fine when using pymssql. However, all databases are currently configured with pyodbc. Do you have any recommendations to work around this issue?
### Screenshots/recordings
### Superset version
4.0.1
### Python version
3.11
### Node version
I don't know
### Browser
Chrome
### Additional context
_No response_
### Checklist
- [X] I have searched Superset docs and Slack and didn't find a solution to my problem.
- [X] I have searched the GitHub issue tracker and didn't find a similar bug report.
- [ ] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] MSSQL Error: list index out of range [superset]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #29494:
URL: https://github.com/apache/superset/issues/29494#issuecomment-2209547177
<!-- Answer -->
To work around the `list index out of range` issue when upgrading from Superset `version 2.1.3` to `version 4.0.2` while using the MSSQL engine with `pyodbc version 5.0.1`, you can modify the `fetch_data` method in the `MssqlEngineSpec` class to handle the unpacking of `pyodbc.Row` objects correctly.
Here is the relevant code snippet with the necessary modification:
```python
@classmethod
def fetch_data(
cls, cursor: Any, limit: Optional[int] = None
) -> list[tuple[Any, ...]]:
if not cursor.description:
return []
data = super().fetch_data(cursor, limit)
# Lists of `pyodbc.Row` need to be unpacked further
return cls.pyodbc_rows_to_tuples(data)
```
Ensure that the `pyodbc_rows_to_tuples` method is correctly implemented to handle the conversion of `pyodbc.Row` objects to tuples. This should help in resolving the `list index out of range` issue <sup>[[1]](https://github.com/apache/superset/blob/master/superset/db_engine_specs/mssql.py)</sup>.
<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] MSSQL Error: list index out of range [superset]
Posted by "Habeeb556 (via GitHub)" <gi...@apache.org>.
Habeeb556 commented on issue #29494:
URL: https://github.com/apache/superset/issues/29494#issuecomment-2212435858
++ Traceback @dosu
```
Traceback (most recent call last):
File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/flask/app.py", line 1484, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/flask/app.py", line 1469, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/flask_appbuilder/security/decorators.py", line 95, in wraps
return f(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/views/base_api.py", line 127, in wraps
raise ex
File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/views/base_api.py", line 121, in wraps
duration, response = time_function(f, self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/utils/core.py", line 1470, in time_function
response = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/views/base_api.py", line 93, in wraps
return f(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/utils/log.py", line 255, in wrapper
value = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/sqllab/api.py", line 409, in execute_sql_query
command_result: CommandResult = command.run()
^^^^^^^^^^^^^
File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/commands/sql_lab/execute.py", line 121, in run
raise ex
File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/commands/sql_lab/execute.py", line 103, in run
status = self._run_sql_json_exec_from_scratch()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/commands/sql_lab/execute.py", line 163, in _run_sql_json_exec_from_scratch
raise ex
File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/commands/sql_lab/execute.py", line 158, in _run_sql_json_exec_from_scratch
return self._sql_json_executor.execute(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/sqllab/sql_json_executer.py", line 111, in execute
raise SupersetErrorsException(
superset.exceptions.SupersetErrorsException: [SupersetError(message='list index out of range', error_type=<SupersetErrorType.GENERIC_DB_ENGINE_ERROR: 'GENERIC_DB_ENGINE_ERROR'>, level=<ErrorLevel.ERROR: 'error'>, extra={'engine_name': 'Microsoft SQL Server', 'issue_codes': [{'code': 1002, 'message': 'Issue 1002 - The database returned an unexpected error.'}]})]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] MSSQL Error: list index out of range [superset]
Posted by "Habeeb556 (via GitHub)" <gi...@apache.org>.
Habeeb556 commented on issue #29494:
URL: https://github.com/apache/superset/issues/29494#issuecomment-2212457306
Okay, I found the error. It was a syntax mistake caused by a small detail in the copied source code. The old version of Python ignored it during execution, but the latest version doesn't, resulting in a `list index out of range` generic error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] MSSQL Error: list index out of range [superset]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #29494:
URL: https://github.com/apache/superset/issues/29494#issuecomment-2211334837
<!-- Answer -->
If the `pyodbc_rows_to_tuples` method is correctly implemented and you are still encountering the `list index out of range` issue, it might be due to changes in the data structure or the way data is fetched in the newer version of Superset. Here are a few additional steps you can take to diagnose and potentially resolve the issue:
1. **Double-Check the Implementation**:
Ensure that the `pyodbc_rows_to_tuples` method is implemented as follows:
```python
@classmethod
def pyodbc_rows_to_tuples(cls, data: list) -> list[tuple]:
return [tuple(row) for row in data]
```
2. **Review the `fetch_data` Method**:
Verify that the `fetch_data` method in the `MssqlEngineSpec` class is correctly calling `pyodbc_rows_to_tuples`:
```python
@classmethod
def fetch_data(
cls, cursor: Any, limit: Optional[int] = None
) -> list[tuple[Any, ...]]:
if not cursor.description:
return []
data = super().fetch_data(cursor, limit)
return cls.pyodbc_rows_to_tuples(data)
```
3. **Check for Breaking Changes**:
Review the release notes for Superset versions between 2.1.3 and 4.0.2 for any breaking changes that might affect the MSSQL engine or the `fetch_data` method. Specifically, the following fix is relevant:
- [#27577](https://github.com/apache/superset/pull/27577) fix: sqlglot SQL Server (@betodealmeida) <sup>[[1]](https://github.com/apache/superset/blob/master/CHANGELOG/4.0.0.md)</sup>.
4. **Add Logging for Debugging**:
Add logging to the `fetch_data` method to capture more details about the data being processed. This can help identify where the `list index out of range` error is occurring:
```python
import logging
logger = logging.getLogger(__name__)
@classmethod
def fetch_data(
cls, cursor: Any, limit: Optional[int] = None
) -> list[tuple[Any, ...]]:
if not cursor.description:
return []
data = super().fetch_data(cursor, limit)
logger.debug("Fetched data: %s", data)
converted_data = cls.pyodbc_rows_to_tuples(data)
logger.debug("Converted data: %s", converted_data)
return converted_data
```
5. **Review Unit Tests**:
Ensure that the unit tests for the `fetch_data` method are passing and cover the scenarios you are encountering. The `test_fetch_data` method in the `tests/unit_tests/db_engine_specs/test_mssql.py` file can provide insights into how the method is expected to behave <sup>[[2]](https://github.com/apache/superset/blob/master/tests/unit_tests/db_engine_specs/test_mssql.py)</sup>.
By following these steps, you should be able to gather more information about the root cause of the `list index out of range` error and identify potential solutions.
<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] MSSQL Error: list index out of range [superset]
Posted by "Habeeb556 (via GitHub)" <gi...@apache.org>.
Habeeb556 closed issue #29494: MSSQL Error: list index out of range
URL: https://github.com/apache/superset/issues/29494
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] MSSQL Error: list index out of range [superset]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #29494:
URL: https://github.com/apache/superset/issues/29494#issuecomment-2212437507
<!-- Answer -->
The traceback indicates that the `list index out of range` error is being raised within the `execute` method of the `SynchronousSqlJsonExecutor` class in `superset/sqllab/sql_json_executer.py`. This method is part of the new SQL query execution logic introduced between Superset versions 2.1.3 and 4.0.2.
Given the traceback, here are some steps to further diagnose and resolve the issue:
1. **Review the `execute` Method**:
Ensure that the `execute` method in the `SynchronousSqlJsonExecutor` class is correctly handling the data fetched from the database. Add logging to capture the state of the data at various points in the method.
```python
import logging
logger = logging.getLogger(__name__)
class SynchronousSqlJsonExecutor:
# Existing attributes and methods
def execute(self, execution_context: SqlJsonExecutionContext, rendered_query: str, log_params: dict[str, Any] | None) -> SqlJsonExecutionStatus:
try:
# Add logging before and after fetching data
logger.debug("Executing query: %s", rendered_query)
data = self._fetch_data(execution_context, rendered_query)
logger.debug("Fetched data: %s", data)
# Process data
# ...
except Exception as ex:
logger.error("Error executing query: %s", ex)
raise
```
2. **Check Data Fetching Logic**:
Verify that the `_fetch_data` method (or equivalent) in the `SynchronousSqlJsonExecutor` class is correctly fetching and processing the data. Ensure that it handles `pyodbc.Row` objects properly.
3. **Review Changes in `superset/commands/sql_lab/execute.py`**:
Examine the changes in the `superset/commands/sql_lab/execute.py` file between versions 2.1.3 and 4.0.2 to identify any modifications that could affect data fetching and processing.
4. **Unit Tests**:
Ensure that unit tests for the `SynchronousSqlJsonExecutor` class and related methods are passing and cover the scenarios you are encountering. This can help identify any discrepancies in the expected behavior.
By following these steps, you should be able to gather more information about the root cause of the `list index out of range` error and identify potential solutions.
<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] MSSQL Error: list index out of range [superset]
Posted by "Habeeb556 (via GitHub)" <gi...@apache.org>.
Habeeb556 commented on issue #29494:
URL: https://github.com/apache/superset/issues/29494#issuecomment-2211299763
@dosu I am sure that `pyodbc_rows_to_tuples` method is correctly implemented and still get the `list index out of range` issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org