Re: [hatari-devel] GEMDOS filename handling

[Eero Tamminen <oak@xxxxxxxxxxxxxx>]

Hi,

On torstai 31 heinäkuu 2014, Max Böhm wrote:
> 2014-07-30 18:39 GMT+02:00 Eero Tamminen <oak@xxxxxxxxxxxxxx>:
> > Thanks, the patch looks good except for few trivial things:
> > - most of the new functions are used only in str.c so they
> > 
> >   could be static
> 
> this is done
> 
> > - Str_HostToAtari() call could be after string has been
> > 
> >   clipped to 8+3 chars and replace "strcpy(dst, src);"
> 
> This couldn't be done as in UTF-8 encoding the host path may take more
> than one byte per character.
> Clipping the hostpath on 8+3 bytes may clip too much.

Ah, right.

> The updated patch is here:
> https://gist.github.com/bfc53cd886b204ea22d8

Thanks!


> > This is good start, but the main point of emulation is to match
> > the behavior of emulated system.  You need to repeat the same test
> > also for different TOS versions [1] on a floppy or HD image.
> > Could you do that testing too?
> > 
> > If behaviour differs between TOS versions, we need to think
> > whether differences should be emulated or whether we just pick
> > the safest behavior.
> > 
> > [1] Latest EmuTOS, TOS 1.0x, TOS 1.6x, TOS 2.0x, TOS 3.x, TOS 4.x.
> > 
> > [2] number of files on FAT file system root dir is limited
> >     (and max path+file name length is 255), so run test in
> >     a sub directory.
> 
> I've repeated the tests (1st and 2nd) under Linux for the TOS versions
> "latest EmuTOS", 1.04, 1.62, 2.06, 3.06, 4.04 using GEMDOS HD emulation.
> In addition I did the test (1st and 2nd) on an ACSI harddisk image.
>
> 1st Test: create files for all characters and then delete them by
> specifying their full name.
> 2nd Test: create files for all characters and then delete them
> manually using the GEM Desktop.
> 
> The results are identical for all tested TOS versions. All created
> files can be deleted in both cases.
>
> The GEMDOS HD emulation can't create files with '\' or '/' in their name.
> The real TOS can't create files with '\' in their name.

Excellent, thanks!


>>> You find the source code of the test program here:
>>> https://gist.github.com/6c7c4340e2a656b2066b
>>
>> Good, this really needed a test-case.
 
But I just noticed that your tester tests just normal files.
Could you add directory creation and removal to it too and
check how those work with real TOS on ACSI or floppy image?

(If you don't have time for that, I could have time for it
in few weeks.)


> The test results for the GEMDOS HD emulation are here:
> https://gist.github.com/9f484d68dd76208adb86
> 
> The test result for the harddisk image is here:
> https://gist.github.com/5b94df54442d7b44bbe3
....
> >> I noticed that the GEMDOS emulation layer can create files on the host
> >> with certain characters in the pathname which cannot be found (by
> >> Fsfirst) or deleted (by Fdelete) thereafter.
> >> 
> >> Characters on Windows:    " * / : < > ? \ |
> >> Characters on Linux/OSX:  / \
> >> 
> >> I assume this is intentional.
> > 
> > Str_Filename2TOSname() filters characters that are invalid for Atari,
> > when emulation populates DTA with host names for the Fsnext() call.
> > I.e. it's for host->atari direction.
> > 
> > For atari->host direction, when emulated program itself specifies file
> > name e.g. for Fcreate(), only thing done to such file names is clipping
> > it to 8+3 characters.  Clipping is needed because some Atari programs
> > give longer strings (e.g. because 8+3 long file names in binary aren't
> > separated by terminating zeros) and real TOS clips them.
> > 
> > The clipping is done in gemdos.c::clip_to_83().  I guess it could be
> > moved to str.c, some host character filtering could be added to it,
> > and the function renamed e.g. Str_Filename2Host().
> > 
> > I think this should be a separate patch.
> > 
> >> But one other behaviour of the GEMDOS emulation is not fully clear to
> >> me. The GEMDOS layer replaces certain characters by '+' when returned
> >> by Fsfirst/Fsnext:
> >> 
> >> ' * . : ? { } 0x7F
> >> 
> >> I don't understand why this is done. As a result those files are not
> >> shown with their original character in the emulated system
> >> 
> >> and you get an error when you try to drag the file into another folder
> >> or into the Trash icon.
> >> 
> >> I know the GEMDOS layer had a wildcard '?' inserted to catch such
> >> cases, but this would match on other files too, which is why I
> >> commented it out. In my view with my patch the replacement of special
> >> characters by '+' would no longer be needed at all, what do you
> >> think?
> > 
> > There can be exactly one '.' in TOS filenames, it's only the rest
> > of '.' characters which are filtered out.
> > 
> > A lot of things could break if you would pass multiple '.' chars
> > through Fsnext().  Some Atari programs expect there to be max 3 chars
> > after first '.', some after last '.'.  This filtering is mainly for
> > files
> > 
> > coming from elsewhere, things like:
> >   this.is.file.with.lots.of.dots.txt
> > 
> > If you want to remove host->atari filtering for the other characters
> > and replace '?' matching for '+' INVALID_CHAR with "[.+]" pattern,
> > I'm fine with that *if* it's tested to work with following:
> > 1. GEM desktop and file selectors in different TOS versions [1],
> >    whether all files show up fine in them and can be selected
> > 
> > 2. several other things using and handling Fsnext() results:
> >    - replacement file selectors (boxkite, slectric...)
> >    
> >    - programs listing and handling file name patterns:
> >      GUI copiers, compressors (egale, twoinone, arcshl,
> >      lzhshell, pacshell, stzip...) etc
> 
> I currently don't have the time to go deeper into this topic and all
> these required tests...

As long as you've done the Atari TOS checks, I can do the additional
testing for Atari programs.  Unfortunately I'm not having much time for
Hatari currently (hopefully more in a week or two), that's why I've been
asking you to do more.


> For me the main question is what to do with the files which receive
> the '+' replacement
> character in Fsfirst/Fsnext. Shall such returned paths be found on the
> host like it currently is (by replacing th '+' by a wildcard '?' in the
> GEMDOS HD layer). Then they can be deleted but the '?' wildcard can also
> match on other files and delete them instead.
> 
> Or shall they not be deletable unless their full name is specified
> without wildcards
> or replacement character? This is what I originally had in my patch,
> but now have
> removed it as you prefered to keep this separate. I think it may need
> more discussion.

The other alternatives are:
- instead of '?', using glob() pattern with '[]', which contains all
  the invalid characters (after your 8-bit char patch, amount of invalid
  chars is small enough to be handled like that).
- or first trying with INVALID_CHAR and if that doesn't match, try with '?'


> I found something else which may need attention: When you create a file
> "atari.log" through the GEMDOC HD layer on the host and then copy another
> file "atari.lo" into the same directory, the two files seem to be mapped
> to the same host file by the GEMDOS layer. The emulated system reports
> that "atari.lo" already exists although this should not be the case.

Ok, I'll take a look at that later on.

If you're interested to check what happens right away, "--trace gemdos"
and setting DEBUG_PATTERN_MATCH to 1 in gemdos.c give quite a bit of
useful debug output.


	- Eero