Deprecate non HX_SMART_STRINGS paths

[Aidan63]

Having to keep two code paths for text encoding stuff has been annoying me recently, I have two questions.

1. What is the NO_HX_SMART_STRINGS really for? I'm assuming this was just to switch back to the old code in case there were bugs in the new dual encoding system.
2. Is anyone actually using the non smart strings mode, if so, why? I'm assuming if anyone is they ran into a bug, disabled unicode string, and haven't touched it since.

It would be nice if for Haxe 5 we could stop supporting the fallback mode, any new api wouldn't need to support both that way.

[tobil4sk]

We could also repurpose these code paths to have a utf-8 only build which is desirable for some cases.

If it does just get removed, it should happen at the same time as dropping haxe 3 support. I think that should happen later because for now there isn't a stable hxcpp version people could be directed to instead. (I still think it would be good to have a separate haxe5 branch for breaking changes, so that backwards compatible bug fixes could still be released/used without having to be detangled).

[Aidan63]

While a pure utf8 path could be useful I think we'd still have the HX_SMART_STRING conditionals sprinkled around which is what I really want to avoid. I agree that we wouldn't want to remove these code paths until we've got one last 4.x release out and are moving towards a 5.x release.

[tobil4sk]

I think there is little benefit to removing the non-HX_SMART_STRING paths, because most of the complexity comes from the runtime checks and having to support both ascii and utf-16 encodings everywhere. This is also the source of most conversion bugs (e.g. #1375) rather than the preprocessor conditions which don't add as much complexity in comparison.

Imo the better path forward would just be to deprecate the smart strings feature itself and to use utf-8 universally. That would solve the preprocessor conditions, and also the runtime conditions, and greatly reduce maintenance complexity for string apis. Removing it would also allow us to link one build of pcre instead of having to build both the 8 bit and 16 bit builds.

[Aidan63]

I just re-read my comment and it seems like I missed out a word and it sounds like I'm wanting to keep both paths. I agree we should only have one for all the reasons you mentioned.
All utf8 is fine by me, it would be nice to know why the smart strings approach was taken in the first place though. Can't remember the exact date but utf8 was pretty ubiquitous when smart strings were added.

[tobil4sk]

I found this blog post from when unicode was added to haxe: https://haxe.org/blog/unicode/

For targets where we handle the string implementation ourselves (hxcpp, HashLink, Eval), we decided to use UCS-2 encoding.

So it sounds like UCS-2 was the originally intended encoding rather than UTF-16. The benefits listed about constant time indexing only apply to UCS-2 and not UTF-16. UCS-2 is obsolete now, and support for the full unicode range is standard and expected so these arguments about UCS-2 don't really apply anymore. Supporting full UTF-16 brings back the variable length problems that are mentioned with UTF-8.

I guess the intent behind a hybrid approach with both encodings was to save memory on text where only ASCII is used, while also supporting full unicode when needed. It seems like Java does a similar thing: https://openjdk.org/jeps/254