Re: [hatari-devel] Too many (4096) active GEMDOS HD DTA entries

[Uwe Seimet <Uwe.Seimet@xxxxxxxxx>]

Hi Eero,

Yes, I was copying quite a large number of files from GEMDOS drives to
various partitions on an image file. Several 1000 files I guess.

Best regards

Uwe

> Hi Uwe,
> 
> On 24.9.2024 20.32, Uwe Seimet wrote:
> > While copying some data with KOBOLD3 Hatari displayed this warning:
> > 
> > WARN : Too many (4096) active GEMDOS HD DTA entries, wrapping DTA index
> 
> TOS has no "forget DTA" OS call, so Hatari has no way of knowing which 
> DTAs' data can be dropped.
> 
> Therefore above warning just means that programs running under Hatari 
> emulation have called FsFirst() OS-call 4k times with different DTA 
> addresses, since emulation was booted.
> 
> Did you run Kobold, or other heavy DTA using programs, several times 
> before this warning?
> 
> 
> > May I assume that the copied data are correct, or are they broken?
> 
> If Kobold actually does FsNext() with a DTA on which it did FsFirst() 
> over *four thousand* Fsetdta() / FsFirst() calls ago, then the filenames 
> associated with DTA would have gone after that warning.
> 
> (Hatari searches for all the matching files on FsFirst() call, and 
> caches them to internal DTA cache entry for FsNext() calls.)
> 
> It seems rather unlikely that Kobold would actively use that many DTAs 
> though, but attached is a patch that allows DTA cache to grow unlimited.
> 
> You could use that to check how many DTAs Kobold copy operation actually 
> consumes...
> 
> (DTA cache increments are logged every 256 entries.)
> 
> 
> 	- Eero

> From 6bf72831f6c470474047fa439d06e670422da3d7 Mon Sep 17 00:00:00 2001
> From: Eero Tamminen <oak@xxxxxxxxxxxxxx>
> Date: Wed, 25 Sep 2024 00:49:00 +0300
> Subject: [PATCH] GEMDOS HD: allow unlimited DTA cache growth
> 
> There's no "forget DTA" OS call, and no mechanism to detect that
> process is not using DTA any more.  If user keeps doing things that
> cause more FsFirst() invocations with new DTAs, more DTA (cache)
> entries are consumed.  Let it grow, and just warn user when more is
> allocated.
> 
> PS. Marking current DTAs as free on program termination, and
> compacting the cache may not work either, in case DTA has been passed
> to another program, or DTA was created by other code (ACC, TOS
> desktop, some other resident program) while foreground program was
> running.
> ---
>  src/gemdos.c | 26 +++++++++-----------------
>  1 file changed, 9 insertions(+), 17 deletions(-)
> 
> diff --git a/src/gemdos.c b/src/gemdos.c
> index 38db359e..bf5c00dd 100644
> --- a/src/gemdos.c
> +++ b/src/gemdos.c
> @@ -110,7 +110,6 @@ typedef enum {
>  
>  #define DTA_MAGIC_NUMBER  0x12983476
>  #define DTA_CACHE_INC     256      /* DTA cache initial and increment size (grows on demand) */
> -#define DTA_CACHE_MAX     4096     /* max DTA cache size (multiple of DTA_CACHE_INC) */
>  
>  #define  BASE_FILEHANDLE     64    /* Our emulation handles - MUST not be valid TOS ones, but MUST be <256 */
>  #define  MAX_FILE_HANDLES    64    /* We can allow 64 files open at once */
> @@ -3130,26 +3129,19 @@ static bool GemDOS_SFirst(uint32_t Params)
>  
>  	if (++DTAIndex >= DTACount)
>  	{
> -		if (DTACount < DTA_CACHE_MAX)
> +		INTERNAL_DTA *pNewIntDTAs;
> +		/* increase DTA cache size */
> +		pNewIntDTAs = realloc(InternalDTAs, (DTACount + DTA_CACHE_INC) * sizeof(*InternalDTAs));
> +		if (pNewIntDTAs)
>  		{
> -			INTERNAL_DTA *pNewIntDTAs;
> -			/* increase DTA cache size */
> -			pNewIntDTAs = realloc(InternalDTAs, (DTACount + DTA_CACHE_INC) * sizeof(*InternalDTAs));
> -			if (pNewIntDTAs)
> -			{
> -				InternalDTAs = pNewIntDTAs;
> -				memset(InternalDTAs + DTACount, 0, DTA_CACHE_INC * sizeof(*InternalDTAs));
> -				DTACount += DTA_CACHE_INC;
> -			}
> -			else
> -			{
> -				Log_Printf(LOG_WARN, "Failed to alloc GEMDOS HD DTA entries, wrapping DTA index\n");
> -				DTAIndex = 0;
> -			}
> +			InternalDTAs = pNewIntDTAs;
> +			memset(InternalDTAs + DTACount, 0, DTA_CACHE_INC * sizeof(*InternalDTAs));
> +			DTACount += DTA_CACHE_INC;
> +			Log_Printf(LOG_WARN, "Increased GEMDOS HD DTA cache to %d entries\n", DTACount);
>  		}
>  		else
>  		{
> -			Log_Printf(LOG_WARN, "Too many (%d) active GEMDOS HD DTA entries, wrapping DTA index\n", DTAIndex);
> +			Log_Printf(LOG_WARN, "Failed to alloc GEMDOS HD DTA entries, wrapping DTA index\n");
>  			DTAIndex = 0;
>  		}
>  	}
> -- 
> 2.39.5
>