You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@texera.apache.org by gi...@apache.org on 2026/08/24 07:57:34 UTC
(texera) 01/01: feat(workflow-operator): keep the row when a Hugging Face text cell is empty (#7569)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-7569-6d1330e9b0b86bc3d757f231a38e4f441ee320f6
in repository https://gitbox.apache.org/repos/asf/texera.git
commit 84e2d7d6498cfccdc1dcef33e1a31dbfaf976f2c
Author: Kary Zheng <15...@users.noreply.github.com>
AuthorDate: Mon Aug 24 07:57:21 2026 +0000
feat(workflow-operator): keep the row when a Hugging Face text cell is empty (#7569)
### What changes were proposed in this PR?
Sentiment Analysis, Spam SMS Detection and Text Summarization each
passed their text column straight to a tokenizer or pipeline. An empty
cell arrives as `None`, which transformers rejects with `ValueError: You
need to specify either text or text_target` for the first two and
`ValueError: text input must be of type str ...` for the third, ending
the run.
An empty value is ordinary input here. A blank CSV cell arrives as null,
since univocity returns null for an empty field and
`AttributeTypeUtils.parseField` passes it through by design.
All three now yield the row with their result attributes left empty.
That follows what the Hugging Face inference operator does with a row it
cannot process: `HuggingFaceCodegenBase` appends the error to that row's
results and continues rather than dropping it.
The visualization operators drop such rows instead, with
`dropna(subset=[...])`, but their output is a chart, where a missing row
costs nothing. These three add columns to each input row, so dropping
would take the user's row out of the output along with the value the
model had nothing to say about. Whitespace-only text takes the same
path, for the same reason.
### Any related issues, documentation, discussions?
Closes #7549
### How was this PR tested?
Each of the three specs gains a case asserting the generated Python
guards the empty cell before the value reaches the tokenizer or
pipeline. All three fail on the previous behavior, 79 passed / 3 failed
before the change and 82 / 0 after.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
Co-authored-by: Claude Opus 5 (1M context) <no...@anthropic.com>
---
.../huggingFace/HuggingFaceSentimentAnalysisOpDesc.scala | 11 ++++++++++-
.../huggingFace/HuggingFaceSpamSMSDetectionOpDesc.scala | 11 ++++++++++-
.../huggingFace/HuggingFaceTextSummarizationOpDesc.scala | 7 +++++++
.../HuggingFaceSentimentAnalysisOpDescSpec.scala | 13 +++++++++++++
.../huggingFace/HuggingFaceSpamSMSDetectionOpDescSpec.scala | 13 +++++++++++++
.../HuggingFaceTextSummarizationOpDescSpec.scala | 13 +++++++++++++
6 files changed, 66 insertions(+), 2 deletions(-)
diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceSentimentAnalysisOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceSentimentAnalysisOpDesc.scala
index d60f1f33ea..0a8fb8a894 100644
--- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceSentimentAnalysisOpDesc.scala
+++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceSentimentAnalysisOpDesc.scala
@@ -75,7 +75,16 @@ class HuggingFaceSentimentAnalysisOpDesc extends PythonOperatorDescriptor {
|
| @overrides
| def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]:
- | encoded_input = self.tokenizer(tuple_[$attribute], return_tensors='pt')
+ | text = tuple_[$attribute]
+ | # An empty cell arrives as None, which the tokenizer rejects. Keep the row
+ | # and leave the scores empty rather than ending the run over a value the
+ | # model has nothing to say about.
+ | if text is None or (isinstance(text, str) and not text.strip()):
+ | for label in ($resultAttributePositive, $resultAttributeNeutral, $resultAttributeNegative):
+ | tuple_[label] = None
+ | yield tuple_
+ | return
+ | encoded_input = self.tokenizer(text, return_tensors='pt')
| output = self.model(**encoded_input)
| scores = softmax(output[0][0].detach().numpy())
| ranking = np.argsort(scores)[::-1]
diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceSpamSMSDetectionOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceSpamSMSDetectionOpDesc.scala
index cef9525570..0daa7cd4aa 100644
--- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceSpamSMSDetectionOpDesc.scala
+++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceSpamSMSDetectionOpDesc.scala
@@ -60,7 +60,16 @@ class HuggingFaceSpamSMSDetectionOpDesc extends PythonOperatorDescriptor {
|
| @overrides
| def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]:
- | result = self.pipeline(tuple_[$attribute])[0]
+ | text = tuple_[$attribute]
+ | # An empty cell arrives as None, which the pipeline rejects. Keep the row
+ | # and leave the results empty rather than ending the run over a value the
+ | # model has nothing to say about.
+ | if text is None or (isinstance(text, str) and not text.strip()):
+ | tuple_[$resultAttributeSpam] = None
+ | tuple_[$resultAttributeProbability] = None
+ | yield tuple_
+ | return
+ | result = self.pipeline(text)[0]
| tuple_[$resultAttributeSpam] = (result["label"] == "LABEL_1")
| tuple_[$resultAttributeProbability] = result["score"]
| yield tuple_""".encode
diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceTextSummarizationOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceTextSummarizationOpDesc.scala
index 6b5c5449b4..541bcf46ac 100644
--- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceTextSummarizationOpDesc.scala
+++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceTextSummarizationOpDesc.scala
@@ -58,6 +58,13 @@ class HuggingFaceTextSummarizationOpDesc extends PythonOperatorDescriptor {
| @overrides
| def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]:
| text = tuple_[$attribute]
+ | # An empty cell arrives as None, which the tokenizer rejects. Keep the row
+ | # and leave the summary empty rather than ending the run over a value the
+ | # model has nothing to say about.
+ | if text is None or (isinstance(text, str) and not text.strip()):
+ | tuple_[$resultAttribute] = None
+ | yield tuple_
+ | return
|
| inputs = self.tokenizer([text], padding="max_length", truncation=True, max_length=512, return_tensors="pt")
| input_ids = inputs.input_ids.to(self.device)
diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceSentimentAnalysisOpDescSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceSentimentAnalysisOpDescSpec.scala
index 7902c572cd..c63cbf67af 100644
--- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceSentimentAnalysisOpDescSpec.scala
+++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceSentimentAnalysisOpDescSpec.scala
@@ -107,6 +107,19 @@ class HuggingFaceSentimentAnalysisOpDescSpec extends AnyFlatSpec with Matchers {
code should not include "\"text\"]"
}
+ it should "guard an empty text cell before it reaches the tokenizer" in {
+ val d = configured()
+ val code = d.generatePythonCode()
+
+ // An empty cell arrives as None, and the tokenizer answers it with
+ // `ValueError: You need to specify either text or text_target`, ending the run.
+ val guard = code.linesIterator
+ .find(_.contains("text is None"))
+ .getOrElse(fail("generated code no longer guards an empty text cell"))
+ guard should include("strip()")
+ code.indexOf("text is None") should be < code.indexOf("self.tokenizer(")
+ }
+
"HuggingFaceSentimentAnalysisOpDesc.getPhysicalOp" should
"wire an OpExecWithCode python executor carrying the operator's ports" in {
val d = configured()
diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceSpamSMSDetectionOpDescSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceSpamSMSDetectionOpDescSpec.scala
index 9da5d52782..f7d39d7beb 100644
--- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceSpamSMSDetectionOpDescSpec.scala
+++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceSpamSMSDetectionOpDescSpec.scala
@@ -96,6 +96,19 @@ class HuggingFaceSpamSMSDetectionOpDescSpec extends AnyFlatSpec with Matchers {
carries(code, "score") shouldBe true
}
+ it should "guard an empty text cell before it reaches the pipeline" in {
+ val d = configured()
+ val code = d.generatePythonCode()
+
+ // An empty cell arrives as None, and the pipeline answers it with
+ // `ValueError: You need to specify either text or text_target`, ending the run.
+ val guard = code.linesIterator
+ .find(_.contains("text is None"))
+ .getOrElse(fail("generated code no longer guards an empty text cell"))
+ guard should include("strip()")
+ code.indexOf("text is None") should be < code.indexOf("self.pipeline(")
+ }
+
"HuggingFaceSpamSMSDetectionOpDesc.getPhysicalOp" should
"wire an OpExecWithCode python executor carrying the operator's ports" in {
val d = configured()
diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceTextSummarizationOpDescSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceTextSummarizationOpDescSpec.scala
index 7e62242372..1c4f3e0d8b 100644
--- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceTextSummarizationOpDescSpec.scala
+++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceTextSummarizationOpDescSpec.scala
@@ -101,6 +101,19 @@ class HuggingFaceTextSummarizationOpDescSpec extends AnyFlatSpec with Matchers {
carries(code, "summary") shouldBe true
}
+ it should "guard an empty text cell before it reaches the tokenizer" in {
+ val d = configured()
+ val code = d.generatePythonCode()
+
+ // An empty cell arrives as None, and the tokenizer answers it with
+ // `ValueError: text input must be of type str ...`, ending the run.
+ val guard = code.linesIterator
+ .find(_.contains("text is None"))
+ .getOrElse(fail("generated code no longer guards an empty text cell"))
+ guard should include("strip()")
+ code.indexOf("text is None") should be < code.indexOf("self.tokenizer([text]")
+ }
+
"HuggingFaceTextSummarizationOpDesc.getPhysicalOp" should
"wire an OpExecWithCode python executor carrying the operator's ports" in {
val d = configured()