[I] Image tasks lose candidate labels and can't read chat responses on third-party providers [texera]

["PG1204 (via GitHub)" <gi...@apache.org>]

PG1204 opened a new issue, #7906:
URL: https://github.com/apache/texera/issues/7906

   ### What happened?
   
   When the Hugging Face Inference operator falls back from `hf-inference` to a third-party chat-completions provider, the image tasks degrade in two ways. This is the same bug class as #7195, in `ImageTaskCodegen` instead of `QaRankingCodegen`.
   
   **1. Candidate labels are dropped from the request.** For `zero-shot-image-classification`, `ImageTaskCodegen.payloadPython` (L92-100) builds the native HF pipeline payload:
   
       payload = {"inputs": <base64 image>, "parameters": {"candidate_labels": labels}}
   
   Only `hf-inference` understands that shape. On a third-party provider, `_call_provider` rebuilds the request as an image part plus a text part (`HuggingFaceCodegenBase.scala` L369, L511, L532), where the text is just `prompt_value`. `parameters` is never carried over, so the provider receives an image with no candidate labels: the input that defines the task. The result is a generic caption instead of a classification, with no error reported.
   
   `visual-question-answering` and `document-question-answering` are unaffected here, since their question is `prompt_value`, which the chat branch does send.
   
   **2. Chat responses are written to the result column as raw JSON.** Chat providers reply with `{"choices": [{"message": {"content": ...}}]}`, and three image tasks cannot read that shape in `ImageTaskCodegen.parsePython`:
   
   - `zero-shot-image-classification` (L147) is grouped with three other tasks and always returns `json.dumps(body)`.
   - `visual-question-answering` and `document-question-answering` (L114) return `body.get("answer", json.dumps(body))`, and a chat response has no `answer` key.
   
   So even when the provider answers correctly, the user sees the raw JSON envelope instead of the answer. `image-to-text` and `image-text-to-text` in the same file already handle `choices` correctly, so the fix is to apply that existing idiom to the remaining three tasks.
   
   Both halves need fixing together: correcting only the request still displays raw JSON, and correcting only the response cleanly displays an answer to a question the model was never asked.
   
   The same raw-JSON output appears for `visual-question-answering` and `document-question-answering` whenever they fall back to a chat provider.
   
   ### How to reproduce?
   
   1. Add a Hugging Face Inference operator with task `zero-shot-image-classification`, an image column, and two or more Candidate Labels.
   2. Choose a model whose cheapest provider is not `hf-inference`, or make `hf-inference` fail, so the request goes through `_call_provider`'s chat branch.
   3. Run the workflow. The outbound request carries the image and the prompt text but no candidate labels, and the result column contains a raw chat JSON envelope rather than a label.
   
   ### Version/Branch
   
   1.3.0-incubating-SNAPSHOT (main)
   
   ### Commit Hash (Optional)
   
   _No response_
   
   ### What browsers are you seeing the problem on?
   
   _No response_
   
   ### Relevant log output
   
   ```shell
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Image tasks lose candidate labels and can't read chat responses on third-party providers [texera]

["xuang7 (via GitHub)" <gi...@apache.org>]

xuang7 closed issue #7906: Image tasks lose candidate labels and can't read chat responses on third-party providers
URL: https://github.com/apache/texera/issues/7906


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Image tasks lose candidate labels and can't read chat responses on third-party providers [texera]

["PG1204 (via GitHub)" <gi...@apache.org>]

PG1204 commented on issue #7906:
URL: https://github.com/apache/texera/issues/7906#issuecomment-5391687380

   /take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org