Support parallel encoding to handle super-long text. Parallel encoding splits long text into multiple chunks and encodes them in parallel. This would be useful for tokenizing super-long context.
Support parallel encoding to handle super-long text.
Parallel encoding splits long text into multiple chunks and encodes them in parallel. This would be useful for tokenizing super-long context.