What happened?
(Description corrected after review of #8617 - see the note at the end.)
TextGenCodegen.parsePython extracts the chat response with no guard at all:
return body["choices"][0]["message"]["content"]
QaRankingCodegen.parsePython guards only on key presence, at three sites:
if "choices" in body:
return body["choices"][0]["message"]["content"]
Neither validates what choices actually contains, so unusual-but-valid responses put non-string values into the result column:
{"choices": [{"message": {"content": null}}]} → Python None is written to the cell{"choices": [{"message": {"content": 42}}]} → the number is written to the cell{"choices": [{"message": {"content": [{"type": "text", "text": "hi"}]}}]} → the raw list is written to the cell, rather than the text. Some OpenAI-compatible providers return content in this parts form.
Shapes that raise: an empty choices list, a missing message or content, a non-dict choices, are already handled: _parse_response wraps the branch in except (KeyError, IndexError, TypeError) and falls back to json.dumps(body). So this is not a crash, and the run is never aborted. The gap is that the extraction trusts the shape it is given and can emit a non-string cell value, where the native hf-inference paths beside it degrade cleanly via body.get("answer", json.dumps(body)).
This path is reachable in practice: of the top 100 models by downloads, 45 for text-generation, 37 for question-answering and 72 for sentence-similarity have a live third-party provider, so the chat branch does execute for these tasks.
The equivalent sites in ImageTaskCodegen were raised in review by @Ma77Ball on #7920; these remaining ones are in code merged earlier (#7798) and were left out of that PR to keep the diff focused.
How to reproduce?
Run the operator with text-generation (or any of the QA/ranking tasks) against a model whose cheapest provider is a third-party chat provider, and have that provider return a 200 whose message.content is null, numeric, or a list of parts. The result column receives that value as-is instead of text.
Version/Branch
1.4.0-incubating-SNAPSHOT (main)
What happened?
(Description corrected after review of #8617 - see the note at the end.)
TextGenCodegen.parsePythonextracts the chat response with no guard at all:QaRankingCodegen.parsePythonguards only on key presence, at three sites:Neither validates what
choicesactually contains, so unusual-but-valid responses put non-string values into the result column:{"choices": [{"message": {"content": null}}]}→ PythonNoneis written to the cell{"choices": [{"message": {"content": 42}}]}→ the number is written to the cell{"choices": [{"message": {"content": [{"type": "text", "text": "hi"}]}}]}→ the raw list is written to the cell, rather than the text. Some OpenAI-compatible providers return content in this parts form.Shapes that raise: an empty
choiceslist, a missingmessageorcontent, a non-dictchoices, are already handled:_parse_responsewraps the branch inexcept (KeyError, IndexError, TypeError)and falls back tojson.dumps(body). So this is not a crash, and the run is never aborted. The gap is that the extraction trusts the shape it is given and can emit a non-string cell value, where the nativehf-inferencepaths beside it degrade cleanly viabody.get("answer", json.dumps(body)).This path is reachable in practice: of the top 100 models by downloads, 45 for
text-generation, 37 forquestion-answeringand 72 forsentence-similarityhave a live third-party provider, so the chat branch does execute for these tasks.The equivalent sites in
ImageTaskCodegenwere raised in review by @Ma77Ball on #7920; these remaining ones are in code merged earlier (#7798) and were left out of that PR to keep the diff focused.How to reproduce?
Run the operator with
text-generation(or any of the QA/ranking tasks) against a model whose cheapest provider is a third-party chat provider, and have that provider return a 200 whosemessage.contentis null, numeric, or a list of parts. The result column receives that value as-is instead of text.Version/Branch
1.4.0-incubating-SNAPSHOT (main)