You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "Ma77Ball (via GitHub)" <gi...@apache.org> on 2026/05/02 08:39:20 UTC
[PR] admin-adjustable max columns with a clear overflow message [texera]
Ma77Ball opened a new pull request, #4669:
URL: https://github.com/apache/texera/pull/4669
<!--
Thanks for sending a pull request (PR)! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
[Contributing to Texera](https://github.com/apache/texera/blob/main/CONTRIBUTING.md)
2. Ensure you have added or run the appropriate tests for your PR
3. If the PR is work in progress, mark it a draft on GitHub.
4. Please write your PR title to summarize what this PR proposes, we
are following Conventional Commits style for PR titles as well.
5. Be sure to keep the PR description updated to reflect all changes.
-->
### What changes were proposed in this PR?
<!--
Please clarify what changes you are proposing. The purpose of this section
is to outline the changes. Here are some tips for you:
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
3. If it is a refactoring, clarify what has been changed.
3. It would be helpful to include a before-and-after comparison using
screenshots or GIFs.
4. Please consider writing useful notes for better and faster reviews.
-->
- **Admin-adjustable max columns.** Adds `csv_parser_max_columns` and `result_table_columns_per_batch` to Admin → Settings → Result Panel. The CSV scan operator and the result table read these from `site_settings` at runtime;
defaults live in `default.conf`.
- **Readable overflow error.** When a CSV row exceeds the limit, the operator now throws `RuntimeException("Max columns of N exceeded.")` Instead of letting Univocity's raw stack trace through. Detection looks at the cause
(`ArrayIndexOutOfBoundsException`) and parses the index from both the Java 8 and Java 9+ message formats — needed because Univocity's own hint string is silently dropped on Java 9+.
- **Refactor.** Extracts the repeated "read int/long from `site_settings` with fallback" pattern into `common/dao/SiteSettings`, replacing inline jOOQ in `ConfigResource`, `CSVScanSourceOpExec`, and `DatasetResource`.
### Any related issues, documentation, discussions?
<!--
Please use this section to link other resources if not mentioned already.
1. If this PR fixes an issue, please include `Fixes #1234`, `Resolves #1234`
or `Closes #1234`. If it is only related, simply mention the issue number.
2. If there is design documentation, please add the link.
3. If there is a discussion in the mailing list, please add the link.
-->
### How was this PR tested?
<!--
If tests were added, say they were added here. Or simply mention that if the PR
is tested with existing test cases. Make sure to include/update test cases that
check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how
you tested step by step, ideally copy and paste-able, so that other reviewers can
test and check, and descendants can verify in the future. If tests were not added,
please describe why they were not added and/or why it was difficult to add.
-->
- New `CSVScanSourceOpExecSpec`: happy path, end-of-input, overflow translation against a real parser, and `isColumnOverflow` cases for both AIOOBE message formats.
- Expanded `admin-settings.component.spec.ts` and `result-table-frame.component.spec.ts` for load/save/reset and the columnLimit fallback chain.
- `sbt Test/compile` clean across DAO, ConfigService, WorkflowOperator, FileService. Frontend specs need Node ≥ 20.19 to run.
- Manual: lowered the limit, ran a workflow on a CSV that exceeded it, confirmed the result panel shows `Max columns of N exceeded.`
### Was this PR authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please include the phrase: 'Generated-by: ' followed by the name of the tool
and its version. If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
Co-authored with: Claude Opus 4.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan commented on PR #4669:
URL: https://github.com/apache/texera/pull/4669#issuecomment-4364961176
> > > > Why do we want to make this Admin-adjustable? I don't feel this makes much sense.
>
> > >
>
> > >
>
> > > >
>
> > >
>
> > >
>
> > > > It should be either fixed at a large number or user-adjustable.
>
> > >
>
> > >
>
> > > We want this parameter to be adjustable by the admins. I am OK with fixing it.
>
> >
>
> > Let's make it a large fixed number, unless there is a disadvantage for that.
>
>
>
> The disadvantage is that univocity allocates memory per parser based on that number (10,000 is negligible, but something like 1,000,000 is not when you want to scale the platform).
This makes sense to me now. Why do we also need batch size adjustable for result table? Can we just pick a fixed number?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "Ma77Ball (via GitHub)" <gi...@apache.org>.
Ma77Ball commented on PR #4669:
URL: https://github.com/apache/texera/pull/4669#issuecomment-4364596838
> #3825 is already closed?
That PR was closed after we decided in a meeting to keep it static, as @aglinxinyuan suggested (just not that large). The problem is that users are hitting that limit and need it increased.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #4669:
URL: https://github.com/apache/texera/pull/4669#issuecomment-4363577894
https://github.com/apache/texera/issues/3825 is already closed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan commented on PR #4669:
URL: https://github.com/apache/texera/pull/4669#issuecomment-4394772455
Please fix the frontend format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "Ma77Ball (via GitHub)" <gi...@apache.org>.
Ma77Ball commented on PR #4669:
URL: https://github.com/apache/texera/pull/4669#issuecomment-4364568745
> > > Why do we want to make this Admin-adjustable? I don't feel this makes much sense.
> >
> >
> > >
> >
> >
> > > It should be either fixed at a large number or user-adjustable.
> >
> >
> > We want this parameter to be adjustable by the admins. I am OK with fixing it.
>
> Let's make it a large fixed number, unless there is a disadvantage for that.
The disadvantage is that univocity allocates memory per parser based on that number (10,000 is negligible, but something like 1,000,000 is not when you want to scale the platform).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "Ma77Ball (via GitHub)" <gi...@apache.org>.
Ma77Ball commented on PR #4669:
URL: https://github.com/apache/texera/pull/4669#issuecomment-4365050421
> > > > > Why do we want to make this Admin-adjustable? I don't feel this makes much sense.
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > > > It should be either fixed at a large number or user-adjustable.
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > > We want this parameter to be adjustable by the admins. I am OK with fixing it.
> >
> >
> > >
> >
> >
> > > Let's make it a large fixed number, unless there is a disadvantage for that.
> >
> >
> > The disadvantage is that univocity allocates memory per parser based on that number (10,000 is negligible, but something like 1,000,000 is not when you want to scale the platform).
>
> This makes sense to me now. Why do we also need batch size adjustable for result table? Can we just pick a fixed number?
That setting I added made the result panel settings menu seem a little empty, and I thought it might be a nice feature for users. I can remove it if it should be in a different PR or if it's not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan merged PR #4669:
URL: https://github.com/apache/texera/pull/4669
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #4669:
URL: https://github.com/apache/texera/pull/4669#issuecomment-4364065538
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/4669?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `53.33333%` with `14 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 38.16%. Comparing base ([`7811115`](https://app.codecov.io/gh/apache/texera/commit/7811115ade6172cf9b7675f223174c54816fe246?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`325a63f`](https://app.codecov.io/gh/apache/texera/commit/325a63f3307a8f7097ddb6131ae50b86ed813a7f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/texera/pull/4669?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...operator/source/scan/csv/CSVScanSourceOpExec.scala](https://app.codecov.io/gh/apache/texera/pull/4669?src=pr&el=tree&filepath=common%2Fworkflow-operator%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Ftexera%2Famber%2Foperator%2Fsource%2Fscan%2Fcsv%2FCSVScanSourceOpExec.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3dvcmtmbG93LW9wZXJhdG9yL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvdGV4ZXJhL2FtYmVyL29wZXJhdG9yL3NvdXJjZS9zY2FuL2Nzdi9DU1ZTY2FuU291cmNlT3BFeGVjLnNjYWxh) | 64.70% | [4 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/texera/pull/4669?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ain/scala/org/apache/texera/dao/SiteSettings.scala](https://app.codecov.io/gh/apache/texera/pull/4669?src=pr&el=tree&filepath=common%2Fdao%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Ftexera%2Fdao%2FSiteSettings.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL2Rhby9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL3RleGVyYS9kYW8vU2l0ZVNldHRpbmdzLnNjYWxh) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/texera/pull/4669?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...pache/texera/service/resource/ConfigResource.scala](https://app.codecov.io/gh/apache/texera/pull/4669?src=pr&el=tree&filepath=config-service%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Ftexera%2Fservice%2Fresource%2FConfigResource.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29uZmlnLXNlcnZpY2Uvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS90ZXhlcmEvc2VydmljZS9yZXNvdXJjZS9Db25maWdSZXNvdXJjZS5zY2FsYQ==) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/texera/pull/4669?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (7811115) and HEAD (325a63f). Click for more details.
>
> <details><summary>HEAD has 3 uploads less than BASE</summary>
>
>| Flag | BASE (7811115) | HEAD (325a63f) |
>|------|------|------|
>|agent-service|1|0|
>|python|1|0|
>|frontend|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #4669 +/- ##
============================================
- Coverage 46.17% 38.16% -8.01%
+ Complexity 1999 1994 -5
============================================
Files 1013 661 -352
Lines 38165 16846 -21319
Branches 3712 1923 -1789
============================================
- Hits 17623 6430 -11193
+ Misses 19769 9758 -10011
+ Partials 773 658 -115
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/4669/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/4669/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/4669/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [python](https://app.codecov.io/gh/apache/texera/pull/4669/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [scala](https://app.codecov.io/gh/apache/texera/pull/4669/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.16% <53.33%> (-0.04%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/texera/pull/4669?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "Ma77Ball (via GitHub)" <gi...@apache.org>.
Ma77Ball commented on code in PR #4669:
URL: https://github.com/apache/texera/pull/4669#discussion_r3199044678
##########
config-service/src/main/scala/org/apache/texera/service/resource/ConfigResource.scala:
##########
@@ -57,7 +58,10 @@ class ConfigResource {
),
"activeTimeInMinutes" -> GuiConfig.guiWorkflowWorkspaceActiveTimeInMinutes,
"copilotEnabled" -> GuiConfig.guiWorkflowWorkspaceCopilotEnabled,
- "limitColumns" -> GuiConfig.guiWorkflowWorkspaceLimitColumns,
+ "limitColumns" -> SiteSettings.getInt(
+ "result_table_columns_per_batch",
+ GuiConfig.guiWorkflowWorkspaceLimitColumns
+ ),
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan commented on PR #4669:
URL: https://github.com/apache/texera/pull/4669#issuecomment-4364536408
> > Why do we want to make this Admin-adjustable? I don't feel this makes much sense.
>
> >
>
> > It should be either fixed at a large number or user-adjustable.
>
>
>
> We want this parameter to be adjustable by the admins. I am OK with fixing it.
Let's make it a large fixed number, unless there is a disadvantage for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan commented on PR #4669:
URL: https://github.com/apache/texera/pull/4669#issuecomment-4365054212
> > > > > > Why do we want to make this Admin-adjustable? I don't feel this makes much sense.
>
> > >
>
> > >
>
> > > > >
>
> > >
>
> > >
>
> > > > >
>
> > >
>
> > >
>
> > > > > >
>
> > >
>
> > >
>
> > > > >
>
> > >
>
> > >
>
> > > > >
>
> > >
>
> > >
>
> > > > > > It should be either fixed at a large number or user-adjustable.
>
> > >
>
> > >
>
> > > > >
>
> > >
>
> > >
>
> > > > >
>
> > >
>
> > >
>
> > > > > We want this parameter to be adjustable by the admins. I am OK with fixing it.
>
> > >
>
> > >
>
> > > >
>
> > >
>
> > >
>
> > > > Let's make it a large fixed number, unless there is a disadvantage for that.
>
> > >
>
> > >
>
> > > The disadvantage is that univocity allocates memory per parser based on that number (10,000 is negligible, but something like 1,000,000 is not when you want to scale the platform).
>
> >
>
> > This makes sense to me now. Why do we also need batch size adjustable for result table? Can we just pick a fixed number?
>
>
>
> That setting I added because the result panel settings menu seems a little empty, and I thought it might be a nice feature for users. I can remove it if it should be in a different PR or if it's not needed.
Let's don't over complicate the settings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "Ma77Ball (via GitHub)" <gi...@apache.org>.
Ma77Ball commented on PR #4669:
URL: https://github.com/apache/texera/pull/4669#issuecomment-4365743154
@aglinxinyuan, please review again. I removed the other column-per-result-panel setting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "Ma77Ball (via GitHub)" <gi...@apache.org>.
Ma77Ball commented on PR #4669:
URL: https://github.com/apache/texera/pull/4669#issuecomment-4393777430
@aglinxinyuan please review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on PR #4669:
URL: https://github.com/apache/texera/pull/4669#issuecomment-4364041330
> Why do we want to make this Admin-adjustable? I don't feel this makes much sense.
>
> It should be either fixed at a large number or user-adjustable.
We want this parameter to be adjustable by the admins. I am OK with fixing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "Ma77Ball (via GitHub)" <gi...@apache.org>.
Ma77Ball commented on code in PR #4669:
URL: https://github.com/apache/texera/pull/4669#discussion_r3199054911
##########
common/config/src/main/resources/default.conf:
##########
@@ -94,3 +94,8 @@ dataset {
multipart_upload_chunk_size_mib = 50
multipart_upload_chunk_size_mib = ${?DATASET_MULTIPART_UPLOAD_CHUNK_SIZE_MIB}
}
+
+csv {
Review Comment:
done
##########
common/config/src/main/resources/default.conf:
##########
@@ -94,3 +94,8 @@ dataset {
multipart_upload_chunk_size_mib = 50
multipart_upload_chunk_size_mib = ${?DATASET_MULTIPART_UPLOAD_CHUNK_SIZE_MIB}
}
+
+csv {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan commented on code in PR #4669:
URL: https://github.com/apache/texera/pull/4669#discussion_r3199307078
##########
common/config/src/main/resources/default.conf:
##########
@@ -93,4 +93,4 @@ dataset {
# the size of each chunk during the multipart upload of file
multipart_upload_chunk_size_mib = 50
multipart_upload_chunk_size_mib = ${?DATASET_MULTIPART_UPLOAD_CHUNK_SIZE_MIB}
-}
+}
Review Comment:
Missing trailing newline — the file used to end with `
` on main, this PR drops it. Please re-add.
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpExecSpec.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.operator.source.scan.csv
+
+import com.univocity.parsers.common.TextParsingException
+import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
+import org.scalatest.flatspec.AnyFlatSpec
+
+import java.io.StringReader
+
+/**
+ * Verifies the column-overflow translation in [[CSVScanSourceOpExec.parseNextRow]]
+ * — the path that turns a deep Univocity stack trace into a single-sentence message
+ * the workflow user can act on.
+ */
+class CSVScanSourceOpExecSpec extends AnyFlatSpec {
+
+ private def parserWithMaxColumns(max: Int): CsvParser = {
+ val settings = new CsvParserSettings()
+ settings.setMaxColumns(max)
+ settings.setMaxCharsPerColumn(-1)
+ new CsvParser(settings)
+ }
+
+ "parseNextRow" should "return the parsed row when the input is within the column limit" in {
+ val parser = parserWithMaxColumns(10)
+ parser.beginParsing(new StringReader("a,b,c\n"))
+
+ val row = CSVScanSourceOpExec.parseNextRow(parser, 10)
+
+ assert(row.toSeq == Seq("a", "b", "c"))
+ }
+
+ it should "return null at end of input (so the iterator can terminate cleanly)" in {
+ val parser = parserWithMaxColumns(10)
+ parser.beginParsing(new StringReader(""))
+
+ assert(CSVScanSourceOpExec.parseNextRow(parser, 10) == null)
+ }
+
+ it should "translate a column-overflow TextParsingException into a clear user message" in {
+ val maxColumns = 2
+ val parser = parserWithMaxColumns(maxColumns)
+ parser.beginParsing(new StringReader("a,b,c,d,e\n"))
+
+ val ex = intercept[RuntimeException] {
+ CSVScanSourceOpExec.parseNextRow(parser, maxColumns)
+ }
+
+ // The message must mention the configured limit so the user knows what was hit.
+ assert(ex.getMessage.contains(maxColumns.toString))
+ assert(ex.getMessage.toLowerCase.contains("max columns"))
+ assert(ex.getMessage.toLowerCase.contains("exceeded"))
+ // The original Univocity exception is preserved as the cause so developers
+ // can still inspect the underlying parser state if needed.
+ assert(ex.getCause.isInstanceOf[TextParsingException])
+ }
+
+ "isColumnOverflow" should "detect AIOOBE causes from Java 8's plain-integer message" in {
+ val cause = new ArrayIndexOutOfBoundsException("5")
+ val ex = new TextParsingException(null, "wrapper", cause)
+ assert(CSVScanSourceOpExec.isColumnOverflow(ex, maxColumns = 5))
+ assert(!CSVScanSourceOpExec.isColumnOverflow(ex, maxColumns = 6))
+ }
+
+ it should "detect AIOOBE causes from Java 9+'s 'Index N out of bounds for length M' message" in {
+ val cause = new ArrayIndexOutOfBoundsException("Index 5 out of bounds for length 5")
+ val ex = new TextParsingException(null, "wrapper", cause)
+ assert(CSVScanSourceOpExec.isColumnOverflow(ex, maxColumns = 5))
+ assert(!CSVScanSourceOpExec.isColumnOverflow(ex, maxColumns = 6))
+ }
+
+ it should "ignore TextParsingExceptions whose cause is unrelated" in {
+ val unrelated = new TextParsingException(null, "Some other parsing problem")
+ val withDifferentCause =
+ new TextParsingException(null, "wrapper", new IllegalStateException("nope"))
+ assert(!CSVScanSourceOpExec.isColumnOverflow(unrelated, maxColumns = 5))
+ assert(!CSVScanSourceOpExec.isColumnOverflow(withDifferentCause, maxColumns = 5))
Review Comment:
Worth adding a case for `ArrayIndexOutOfBoundsException` with an unparseable message — that's the exact path the `forall → exists` fix protects, and the current "unrelated cause" test only covers `IllegalStateException`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan commented on code in PR #4669:
URL: https://github.com/apache/texera/pull/4669#discussion_r3198736029
##########
config-service/src/main/scala/org/apache/texera/service/resource/ConfigResource.scala:
##########
@@ -57,7 +58,10 @@ class ConfigResource {
),
"activeTimeInMinutes" -> GuiConfig.guiWorkflowWorkspaceActiveTimeInMinutes,
"copilotEnabled" -> GuiConfig.guiWorkflowWorkspaceCopilotEnabled,
- "limitColumns" -> GuiConfig.guiWorkflowWorkspaceLimitColumns,
+ "limitColumns" -> SiteSettings.getInt(
+ "result_table_columns_per_batch",
+ GuiConfig.guiWorkflowWorkspaceLimitColumns
+ ),
Review Comment:
The admin UI no longer writes this key — leftover from the original design. Please revert this hunk back to the direct `GuiConfig.guiWorkflowWorkspaceLimitColumns` read.
##########
common/dao/src/main/scala/org/apache/texera/dao/SiteSettings.scala:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.texera.dao
+
+import org.jooq.impl.DSL
+
+import scala.util.Try
+
+/**
+ * Read-side accessor for the `site_settings` key/value table that admin pages
+ * write through. Centralises the "look up by key, parse, fall back on any
+ * failure" pattern that previously lived inline in ConfigResource,
+ * CSVScanSourceOpExec, and DatasetResource.
+ *
+ * Failures swallowed by the outer Try include: SqlServer not initialised
+ * (e.g. on workers in distributed mode), no row for the key, and value that
+ * can't be parsed. In all of these cases the caller's default takes over.
Review Comment:
0% test coverage. Please extract the parse-or-fallback logic into a pure function and add a unit test.
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpExec.scala:
##########
@@ -106,3 +112,41 @@ class CSVScanSourceOpExec private[csv] (descString: String) extends SourceOperat
}
}
}
+
+object CSVScanSourceOpExec {
+ val DEFAULT_MAX_COLUMNS = 512
+
+ def getMaxColumns: Int =
+ SiteSettings.getInt("csv_parser_max_columns", DEFAULT_MAX_COLUMNS)
+
+ /**
+ * Wraps `parser.parseNext()` so a column-count overflow is reported to the user
+ * as a clear instruction rather than a deep Univocity stack trace. Other parser
+ * failures are rethrown unchanged.
+ *
+ * The thrown RuntimeException's message bubbles up through DataProcessor.handleExecutorException
+ * and becomes the title of the console message that drives the top-of-page toast.
+ */
+ def parseNextRow(parser: CsvParser, maxColumns: Int): Array[String] = {
+ try parser.parseNext()
+ catch {
+ case e: TextParsingException if isColumnOverflow(e, maxColumns) =>
+ throw new RuntimeException(columnOverflowMessage(maxColumns), e)
+ }
+ }
+
+ private[csv] def isColumnOverflow(e: TextParsingException, maxColumns: Int): Boolean =
+ Option(e.getCause)
+ .collect { case aioobe: ArrayIndexOutOfBoundsException => aioobe }
+ .exists(aioobe => aioobeIndex(aioobe).forall(_ == maxColumns))
Review Comment:
`forall` returns true on `None`. If the JDK changes the AIOOBE message format again, an unrelated AIOOBE would be reported as a column-overflow. Use `.exists(_ == maxColumns)`.
##########
common/config/src/main/resources/default.conf:
##########
@@ -94,3 +94,8 @@ dataset {
multipart_upload_chunk_size_mib = 50
multipart_upload_chunk_size_mib = ${?DATASET_MULTIPART_UPLOAD_CHUNK_SIZE_MIB}
}
+
+csv {
Review Comment:
This block is not read anywhere — the default actually comes from the hardcoded `DEFAULT_MAX_COLUMNS = 512` in `CSVScanSourceOpExec`, so `CSV_PARSER_MAX_COLUMNS` has no effect. Either wire it up via a `CsvConfig` object (like `DatasetConfig`) or remove the block.
Also, `csv.csv_parser_max_columns` repeats the namespace; should be `csv.parser_max_columns` to match `dataset.multipart_upload_chunk_size_mib`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on PR #4669:
URL: https://github.com/apache/texera/pull/4669#issuecomment-4365116105
Agreed. Let's take a simple solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: admin-adjustable max columns with a clear overflow message [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on PR #4669:
URL: https://github.com/apache/texera/pull/4669#issuecomment-4364042454
@kunwp1 @SarahAsad23 Feel free to chime in based on a related use case with a collaborator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org