You are viewing a plain text version of this content. The canonical link for it is here.

Posted to notifications@texera.apache.org by "kz930 (via GitHub)" <gi...@apache.org> on 2026/08/11 23:29:42 UTC

[PR] feat(sklearn): skip rows with missing values instead of ending the run [texera]

kz930 opened a new pull request, #7583:
URL: https://github.com/apache/texera/pull/7583

   ### What changes were proposed in this PR?
   
   Every Sklearn operator ended the execution when a cell it reads was empty, with an error raised by scikit-learn's own input validation rather than by the operator: `ValueError: Input X contains NaN.` Nothing in the family looked at missing values, so the message named `X`, a variable inside generated code, and pointed at neither the column nor the row. The training and classifier operators feed every column except the target into the estimator, so a blank in a column that has nothing to do with the model ended the run just the same.
   
   An empty value is ordinary input here. A blank CSV cell arrives as null, since univocity returns null for an empty field and `AttributeTypeUtils.parseField` passes it through by design. Skipping is what the rest of the codebase does with a value that is not there: twenty-four visualization operators open their generated Python with `dropna`, `COUNT(column)` counts only non-null rows, CONCAT and MIN pass over them, and `FilterPredicate` answers false for every condition but IS_NULL / IS_NOT_NULL. This family was the only one with no answer at all.
   
   What skipping means follows what each operator emits.
   
   | Operators | Output | Columns read | Change |
   | --- | --- | --- | --- |
   | `SklearnTrainingOpDesc` subclasses | one row holding the model | every column but the target | drop on the whole table, or on the text and target pair when Count Vectorizer is on |
   | `SklearnClassifierOpDesc` subclasses | one row holding the model | every column but the target, on both ports | the same statement serves both ports |
   | `SklearnTestingOpDesc` | one row per model, with metric columns | every column but the target | drop before scoring |
   | `SklearnAdvancedBaseDesc` subclasses | one row per parameter combination | the named features and the ground truth | drop on those columns only |
   | `SklearnPredictionOpDesc` | each input row, plus a result column | every column but the ground truth | keep the row, leave the result empty |
   
   The last row differs deliberately. Prediction adds a column to the user's rows, so dropping would take the user's row out of the output along with the value the model had nothing to say about. That is what the Hugging Face inference operators already do with a row they cannot process.
   
   Two of the drops name their columns rather than taking the whole table. With Count Vectorizer on, training and classifier read only the text and the target, and the advanced trainers read a named list of features, so a blank in a column they never touch should not cost the row.
   
   Fitting on fewer rows changes the model, the way COUNT over a column with nulls changes the count. That trade is one this codebase has already made everywhere else, and it beats ending the run.
   
   ### Any related issues, documentation, discussions?
   
   Closes #7582
   
   ### How was this PR tested?
   
   Each changed operator gained a case in its existing spec asserting the drop appears in the generated Python, next to the cases already asserting on that output: the two paths for training, the shared statement for the classifier, the scorer's drop, the advanced trainers' named-column drop, and prediction's kept row with an empty result. The full `WorkflowOperator` suite passes, 2192 tests.
   
   The behaviour itself was checked against the reproduction in the issue, a four-row CSV with one blank cell run through `CSV File Scan` into `Training: Bernoulli Naive Bayes`, which ended the execution before this change.
   
   ### Was this PR authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Claude Opus 5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org