Re: [AD] unicode proposal

[Peter Wang <novalazy@xxxxxxxxxx>]

On 2009-01-20, Elias Pschernig <elias.pschernig@xxxxxxxxxx> wrote:
> 
> The conversion functions likely can be useful internally in the
> filesystem code - but the user should not have to deal with anything but
> UTF8 when using A5's filesystem layer.

So the user works with UTF-8 strings and Allegro will automatically
convert to and from the "correct" encoding for the filesystem?

I don't think that will be robust on Unix/Linux where there is not
necessarily _a_ filesystem encoding.  I have filenames in UTF-8 but
occasionally end up with filenames in other (but ASCII-compatible)
encodings.  I usually fix them up immediately but it's only possible
because, say, mv, works properly even when filenames are not in the
expected encoding.  It just treat filenames as raw 8-bit strings.

For Unix, I think that's what we should do as well.  No conversions.
All special characters (e.g. path separators, globbing wildcards) are in
the ASCII range so any ASCII-compatible encoding will work.  That means
we might return filenames to the user which are not UTF-8, and they
might try to display them.  I think they will just have to detect if the
string is not UTF-8, and if so assume an encoding and convert it before
displaying.  We can make that easy for the Latin-1 case.

For Windows, I don't know.  From what I can tell, filenames are stored
in UTF-16 so converting to/from UTF-8 is possible, and probably the
right thing to do so the user only has to deal with ASCII-compatible
filenames everywhere.

Peter