You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@texera.apache.org by "Ma77Ball (via GitHub)" <gi...@apache.org> on 2025/10/04 20:19:15 UTC
[I] Support CSV files with more than 512 columns [texera]
Ma77Ball opened a new issue, #3825:
URL: https://github.com/apache/texera/issues/3825
# Summary
When parsing wide CSV files (e.g., 1000+ columns), Univocity throws an ArrayIndexOutOfBoundsException due to a hard limit of 512 columns. This prevents the ingestion of valid datasets with more columns.
# How to Reproduce
1. Create a CSV file with >512 columns (e.g., 1000 columns × 100 rows).
2. Run the workflow using the CSV Scan Operator.
3. Observe the error in the result panel (shown below)
<img width="1916" height="958" alt="Image" src="https://github.com/user-attachments/assets/989b6805-ac8c-4417-a191-d0d450b62815" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support CSV files with more than 512 columns [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on issue #3825:
URL: https://github.com/apache/texera/issues/3825#issuecomment-3368523319
@Yicong-Huang How to follow the new "type protocol" for this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support CSV files with more than 512 columns [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on issue #3825:
URL: https://github.com/apache/texera/issues/3825#issuecomment-3368524265
@Ma77Ball can you try use the bug template to create it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support CSV files with more than 512 columns [texera]
Posted by "Ma77Ball (via GitHub)" <gi...@apache.org>.
Ma77Ball commented on issue #3825:
URL: https://github.com/apache/texera/issues/3825#issuecomment-3368529176
@Yicong-Huang, I can recreate it using the bug template. I was wondering if this would be considered a bug or a new feature, as I think the goal is to allow the admin to set the maximum number of columns allowed to prevent memory issues, which is similar to why Univocity caps it at 512 by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support CSV files with more than 512 columns [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on issue #3825:
URL: https://github.com/apache/texera/issues/3825#issuecomment-3368530271
I see. It is indeed more of a feature request. Then let's keep it as it is. I will later introduce a feature template. You can ignore the comments regarding bug template.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support CSV files with more than 512 columns [texera]
Posted by "Ma77Ball (via GitHub)" <gi...@apache.org>.
Ma77Ball commented on issue #3825:
URL: https://github.com/apache/texera/issues/3825#issuecomment-3368545363
Currently, the system can process that many columns (as shown below).
<img width="956" height="954" alt="Image" src="https://github.com/user-attachments/assets/202a016e-4d70-4b51-af56-1e90fece81de" />
However, if the output is not passed to an operator that reduces the number of columns, the frontend will crash.
<img width="953" height="983" alt="Image" src="https://github.com/user-attachments/assets/e623134a-07ea-48d7-af13-6521457db10d" />
I'm examining two key potential issues: the result panel is not optimized for this task, and the schema propagation step currently uses 100 rows to evaluate the schema, which may be excessive, especially with larger datasets (1 million columns).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support CSV files with more than 512 columns [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica closed issue #3825: Support CSV files with more than 512 columns
URL: https://github.com/apache/texera/issues/3825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org