You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "nagarajmmu (via GitHub)" <gi...@apache.org> on 2025/07/16 22:11:34 UTC
[I] Chart is not showing the columns, if dataset has zero records [superset]
nagarajmmu opened a new issue, #34196:
URL: https://github.com/apache/superset/issues/34196
### Bug description
Hi All
We are creating a superset dataset and creating a chart from the dataset, we are creating a dataset from the parquet file, using duckdb engine.
When our parquet file is empty (But columns exists) and we are creating the dataset using the empty parquet. We use this dataset to create a chart
Once chart is created, chart will show the error saying that columns are missing.
Because my parquet file is empty, superset is not able to read the column, if parquet/dataset records are zero.
Please help us how we can fix this issue in superset.
Please find the screenshot for referance
<img width="832" height="519" alt="Image" src="https://github.com/user-attachments/assets/6a48d403-691e-4c33-987a-a692a87d168f" />
### Screenshots/recordings
_No response_
### Superset version
4.1.2
### Python version
3.9
### Node version
16
### Browser
Chrome
### Additional context
_No response_
### Checklist
- [ ] I have searched Superset docs and Slack and didn't find a solution to my problem.
- [ ] I have searched the GitHub issue tracker and didn't find a similar bug report.
- [ ] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Chart is not showing the columns, if dataset has zero records [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #34196:
URL: https://github.com/apache/superset/issues/34196#issuecomment-5124484932
Curious whether DuckDB is really returning column metadata for a zero-row parquet source, and if so whether that's something we could pull in during dataset sync instead of relying on a data-bearing query.
This issue has been silent for quite a while... if nobody has theories/tests/repros/PRs, we may close this as not-planned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Chart is not showing the columns, if dataset has zero records [superset]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #34196:
URL: https://github.com/apache/superset/issues/34196#issuecomment-3081378214
<!-- Answer -->
This happens because Superset relies on the database engine (in this case, DuckDB via SQLAlchemy) to provide column metadata. When your Parquet file is empty, DuckDB/SQLAlchemy may not return any columns, so Superset treats the dataset as having no columns, which leads to the error when building charts. This is a known limitation and has been discussed in the community—other users have run into similar issues when syncing columns for datasets that return no data [[discussion](https://github.com/apache/superset/discussions/31573)].
A common workaround is to temporarily add a dummy row to your Parquet file or use a fake query that always returns at least one row, sync the columns in Superset, and then revert to your original (empty) dataset [[issue](https://github.com/apache/superset/issues/33591)]. Superset currently does not have built-in logic to infer or synthesize columns from an empty table; it simply exposes what the database engine reports [[code reference](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/connectors/sqla/models.py)].
If you need to automate this, you could script the addition and removal of a dummy row as part of your data pipeline. Otherwise, keep an eye on updates to DuckDB and Superset, as improvements in schema introspection for empty files may be added in the future.
<!-- Dosu Comment Footer -->
*To reply, just mention [@dosu](https://go.dosu.dev/dosubot).*
---
<sup>How did I do? [Good](https://app.dosu.dev/response-feedback/f326bba0-b530-4f0f-b740-c37e2f61d17e?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/f326bba0-b530-4f0f-b740-c37e2f61d17e?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/f326bba0-b530-4f0f-b740-c37e2f61d17e?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/f326bba0-b530-4f0f-b740-c37e2f61d17e?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/f326bba0-b530-4f0f-b740-c37e2f61d17e?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/f326bba0-b530-4f0f-b740-c37e2f61d17e?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/f326bba0-b530-4f0f-b740-c37e2f61d17e?feedback_type=other)</sup> [](https://go.dosu.dev/discord-bot) [!
[Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/superset/issues/34196)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Chart is not showing the columns, if dataset has zero records [superset]
Posted by "nagarajmmu (via GitHub)" <gi...@apache.org>.
nagarajmmu commented on issue #34196:
URL: https://github.com/apache/superset/issues/34196#issuecomment-3085632724
Hi @rusackas
We are using DUCK DB to get process the data, as we verified DUCKDB is always return the column names, even though data is Zero.
But superset is not able to recognize the column, if data is zero in table.
If superset is able to get the columns form the underlying dataset, we will not face this issue.
Adding dummy records will mislead the user how this data came in to picture, this where we are bit hesitate to use the dummy row when there is zero data.
As of now we are planning to use below query, please let me know, this is recommended to use or also let us know, if this is not the right way to use query.
Query planning to use:
```
SELECT * FROM your_table
WHERE EXISTS (SELECT 1 FROM your_table limit 1)
UNION BY NAME
SELECT NULL as <Existing_Column>
WHERE NOT EXISTS (SELECT 1 FROM your_table limit 1);
```
Thanks
Nagaraj M M
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Chart is not showing the columns, if dataset has zero records [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #34196:
URL: https://github.com/apache/superset/issues/34196#issuecomment-3084812028
I think the bot is right, but paging Dr. @betodealmeida in case they have a better idea of how we can support this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Chart is not showing the columns, if dataset has zero records [superset]
Posted by "Ujjwaljain16 (via GitHub)" <gi...@apache.org>.
Ujjwaljain16 commented on issue #34196:
URL: https://github.com/apache/superset/issues/34196#issuecomment-5584059161
I investigated this against both the current codebase and the historical Superset 4.1.2 stack.
I was able to reproduce the relevant setup with a genuinely empty Parquet file that still contains a valid schema, using the 4.1.2-era dependency versions (`SQLAlchemy 1.4.52`, `duckdb-engine 0.9.5`, and period-correct DuckDB). In the tested cases, DuckDB/Superset preserved the column metadata for both zero-row and non-zero-row sources, including through the metadata paths used by Superset.
I also investigated the previously suspected `duckdb-engine` `cursor.description` behavior and could not establish it as the cause of this issue. The behavior we could reproduce occurs at a different SQLAlchemy layer than the raw DBAPI cursor used by Superset's virtual-dataset metadata path.
At this point I don't have enough evidence to propose a safe code fix for #34196.
Could you provide the following so we can reproduce the original case more precisely?
* A minimal/example Parquet file that reproduces the problem, if possible
* Whether the dataset is a **Physical** or **Virtual** dataset
* The exact DuckDB and `duckdb-engine` versions installed in the environment
* The exact chart error/message shown when the columns are reported as missing
* Whether the columns are visible immediately after creating the dataset, or disappear after a metadata sync/refresh
* If it is a virtual dataset, the SQL/query used to create it
With those details, we should be able to identify where the column metadata is actually being lost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Chart is not showing the columns, if dataset has zero records [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #34196:
URL: https://github.com/apache/superset/issues/34196#issuecomment-5593716025
Went and actually tested this instead of reasoning about it further. A zero-row parquet file that still carries real column types reflects fine in DuckDB, `information_schema` and `DESCRIBE` both return the full column list, and that's the same path Superset's dataset sync goes through. Where it actually breaks is if the source has zero columns, not just zero rows, which is what you get writing an untyped `pd.DataFrame()` straight to parquet. DuckDB can't even open that file.
Nagaraj, your original description says your file has real columns and just no rows, so I'm not sure the zero-column case above is what's actually happening on your end. @Ujjwaljain16 already asked the right questions (sample file, physical vs virtual dataset, duckdb-engine version, whether columns disappear after a sync), so let's hold off closing until one of you can answer those. If you do have a genuine zero-row-with-schema case that Superset's still dropping columns from, that's a real bug worth fixing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org