[AD] Datafile Cache

[VERSTEEGH@xxxxxx]

>Not that I know of.
Good, I have a working system, just need to add the malloc checks ;-)


>> It works as follows:
>>
>> 1 new api function: set_cache_size(int nr_of_files_to_cache)
>>
>I'd prefer to leave this out. Caching the object offsets is an
>implementation detail, and not something that users need to know about. Once
>you add that function, you are constraining all future implementations to
>provide a similar implementation, which might not always be desirable.
Ok, I can completely hide it , but then we need to come up with a sensible
default. Anyway, I'd like to have a way to turn it off, but if the opinion
is that that's not necessary I'll leave it out

>> this will keep a table of contents in memory for the last accessed
>> datafiles this table of contents is used by load_datafile_object, and also
>> built by load_datafile_object so there is hardly any overhead in it.

>Cool. But don't forget about pack_fopen("x.day#y"). The caching needs to
Is'n't pack_fopen_special_file used by load_datafile_object?
I don't use the real file position, but my own counter from the start
of the file (actually from the first object) because I hoped that
would catch all problems with these things.(I fseek relatively from
that point)


>Are you reading and caching the entire index when the file is first touched,
>or only caching objects you pass while looking for the requested one? (the
>latter would be more efficient for isolated calls, but more complex to
>implement).
The latter ofcourse (the first would be less fun implementing ;-) )
It works like this:
When load_datafile_object is called, it will first ask the cache if the object
is cached. If there is no cache at all yet, it will start the normal
searching, meanwhile adding object-adresses to the cache. When the obejct
is found it jumps directly to it.
When there is a cache , but the object is not yet in it, it will start
searching from the last cached position (because you know the object isn't
before that).



>I'm not sure which method would be best, but tend to think maybe
>you should read the entire top level index in one pass, but only look inside
>nested datafiles if the program requests that.
I doubt about nested datafiles at the moment. I'm not sure if it works
right now, it should work if you include them as binary object and use
the parent#child notation as filename.
Does load_datafile_object work for objects in nested datafiles? I never used
that, how would you specify that? I mean, load_datafile_object("parent#child",
"object") would try to read from a datafile included as binary data right?
I'm not sure if it would be worth the trouble to add caching for the indexing
of nested datafiles (it would get quite hairy, while now it's nice&elegant)

Martijn Versteegh