What happened?
File Scan offers an Encoding field. FileScanSourceOpDesc stores it in its own encoding property, but FileScanSourceOpExec decodes with fileEncoding, the property inherited from ScanSourceOpDesc. The class carries @JsonIgnoreProperties(Array("limit", "offset", "fileEncoding")), so fileEncoding never survives serialization into the executor and is always its default, UTF_8.
Choosing any other charset therefore changes nothing. A UTF-16 file comes back decoded as UTF-8 rather than as its text.
Expected: the executor decodes with the charset the Encoding field names.
How to reproduce?
Deserialize a File Scan descriptor carrying "encoding":"UTF_16", write it out the way getPhysicalOp does, and read it back the way FileScanSourceOpExec does. fileEncoding comes back UTF_8, and the encoding the user chose is the only place UTF-16 survives.
In the UI: upload a UTF-16 text file, drop a File Scan on it, set Encoding to UTF_16 and run. The rows hold the file's bytes read as UTF-8, not its lines.
Version/Branch
1.4.0-incubating-SNAPSHOT (main)
Commit Hash (Optional)
2ab8ee0
What browsers are you seeing the problem on?
No response
Relevant log output
JSON = {"operatorType":"FileScan","dummyPropertyList":[],"encoding":"UTF_16","extract":false,"outputFileName":false,"attributeType":"string","attributeName":"line",...}
exec reads desc.fileEncoding = UTF_8
What happened?
File Scan offers an Encoding field.
FileScanSourceOpDescstores it in its ownencodingproperty, butFileScanSourceOpExecdecodes withfileEncoding, the property inherited fromScanSourceOpDesc. The class carries@JsonIgnoreProperties(Array("limit", "offset", "fileEncoding")), sofileEncodingnever survives serialization into the executor and is always its default,UTF_8.Choosing any other charset therefore changes nothing. A UTF-16 file comes back decoded as UTF-8 rather than as its text.
Expected: the executor decodes with the charset the Encoding field names.
How to reproduce?
Deserialize a File Scan descriptor carrying
"encoding":"UTF_16", write it out the waygetPhysicalOpdoes, and read it back the wayFileScanSourceOpExecdoes.fileEncodingcomes backUTF_8, and theencodingthe user chose is the only place UTF-16 survives.In the UI: upload a UTF-16 text file, drop a File Scan on it, set Encoding to UTF_16 and run. The rows hold the file's bytes read as UTF-8, not its lines.
Version/Branch
1.4.0-incubating-SNAPSHOT (main)
Commit Hash (Optional)
2ab8ee0
What browsers are you seeing the problem on?
No response
Relevant log output
JSON = {"operatorType":"FileScan","dummyPropertyList":[],"encoding":"UTF_16","extract":false,"outputFileName":false,"attributeType":"string","attributeName":"line",...} exec reads desc.fileEncoding = UTF_8