Re: [hatari-devel] memory snapshot format/issues (was: Release time for 1.8.1 ?)

[Eero Tamminen <oak@xxxxxxxxxxxxxx>]

Hi,

On perjantai 08 toukokuu 2015, Steven Noonan wrote:
> On Fri, May 8, 2015 at 1:31 PM, Nicolas Pomarède <npomarede@xxxxxxxxxxxx> 
wrote:
>> I thought about using 1.9.0, 1.10.0, 1.11.0, .. too, but as Thomas
>> wrote the version if of the form "x.y.z\0" and it's kinda hardcoded in
>> memorysnapshot.c to take 6 bytes when saving/restoring, so using 1.10.0
>> for example would take 7 bytes in the memory snapshot header, which
>> might not be correctly handled (we need to handle the case where an
>> older Hatari version will load a more recent snapshot and should tell
>> it's not compatible).
>>
>> If we can update the use of VersionString in MemorySnapShot_OpenFile to
>> work with more than 6 bytes of ident string, then I think we can use
>> 1.9.0 for next version, then 1.10.0, 1.11.0, ...

Older version would do strcmp() for strings with lengths of 6 & 7 bytes.
First 6 chars will obviously be different so it doesn't try to load
wrong version.

I would also assume that strcmp() stops at first differing
character, so loading 6 bytes of 7 byte string shouldn't be
a problem in practice.

However, I checked memorysnapShot.c history, and earlier versions
try to print out the on-disk snapshot version, which would now
be missing the terminating NULL, which could cause issues
(at least mess up terminal with non-printable characters).

So, the new format snapshot identification file format should
have '\0' within first 6 characters.


> Hmm, I'm not sure I agree with the "6 vs 7 bytes is a problem"
> argument. Yes, it's a true statement if you are only considering
> snapshot portability between different Hatari versions. I'll grant
> that.
> 
> But I think snapshots really need an overhaul for correctness reasons
> -- for example if you save the memory state in a 64-bit build of
> Hatari and try to load it in a 32-bit build of Hatari, it won't work
> (usually by crashing) because it serialized a bunch of 64-bit pointers
> to disk and the structure offsets of the state on disk are all wrong.

I haven't really looked at this in detail/ages.  Where it's
serializing pointers?


> In my opinion, the whole "fwrite() all your in-memory structures to a
> file" is a poor design.

AFAIK it's primarily meant for snapshots that are used by
the same binary, e.g. when you're actively debugging things.
It's not meant as something portable.


> There are even some issues with saving/restoring snapshots going from
> Hatari 32-bit x86 to Hatari using the Linux x32 ABI (basically full
> x86_64 but with 32-bit pointers). This *should* be totally compatible
> because the pointer sizes are the same, and thus all the structure
> sizes should be identical. However, some of the structure packing is
> different between the two builds which causes memory state
> save/restore to break. I fixed a few of these issues in my local repo,
> but haven't submitted changes to the list.

Please do.


> There are also problems trying to move snapshots between different
> architectures. e.g. 32-bit x86 snapshot will have different endianness
> than a 32-bit PowerPC snapshot. So endianness needs normalization too.

Or Hatari could just reject snapshot if it has wrong endianess.

Having some magic 32-bit value (one with '\0' for older
Hatari versions) as first thing in the file, could
be that check.


> I've been playing with some ideas to use something like protobuf-c or
> msgpack-c for portable and compact memory state save/restore, but I
> haven't touched it in a while. Of the two options, msgpack felt like
> the easiest and simplest approach. The serialization API for msgpack-c
> is quite pleasant and is code size friendly, but the deserialization
> side wasn't nearly as simple just because of the msgpack-c API design.
> Perhaps there's an alternative I haven't looked at yet.

From user's point of view having completely portable memory snapshots
could be interesting, but I think people mostly just want their memory
snapshots of games to work with newer Hatari versions, on the same
HW & OS.  However, that's unlikely to happen as there are changes
between Hatari versions on the data that needs to be stored.


	- Eero