zlib status

SGDK only sub forum

Moderator: Stef

Post Reply
cero
Very interested
Posts: 338
Joined: Mon Nov 30, 2015 1:55 pm

zlib status

Post by cero » Sat Dec 19, 2015 11:38 am

Zlib compresses rather well, for level data, etc.

Using miniz tinfl at -O3, the Genesis decompresses at 23.1 kb/s. It uses about 11kb of RAM. The optimized implementation for the NES uses 1.2kb RAM, but as 11kb is not that much, and the C source was easy to port, I didn't try to reduce the RAM usage.

As the license is permissive, would there be interest in adding it to SGDK?

Stef
Very interested
Posts: 3131
Joined: Thu Nov 30, 2006 9:46 pm
Location: France - Sevres
Contact:

Re: zlib status

Post by Stef » Sat Dec 19, 2015 1:46 pm

SGDK already include some compression schemes, aplib for instance is a multiple purpose. How does miniz compare to this one ?
I would like to add efficient compression/decompression capabilities in SGDK but i don't want to provide too much solutions so developers got completely lost in selecting the good one. Ideally i think 2 generic methods would be enough (1 providing fast unpacking with average compression level and another providing slow unpacking with good compression level).

cero
Very interested
Posts: 338
Joined: Mon Nov 30, 2015 1:55 pm

Re: zlib status

Post by cero » Sat Dec 19, 2015 2:10 pm

aplib is closed source, and therefore I won't use it no matter how good it is. EDIT: This is for level data, and other user things, not for tile compression. It could be used to tile compression too, but I intended it to be exposed so the users can compress game-specific data.

Zlib is also industry standard, installed on practically every computer, while I had never even heard of aplib or uftc before coming here.

That said, zlib compresses better than aplib according to this link:
http://www.maximumcompression.com/data/summary_mf2.php

I did some quick optimizations, dropping the RAM use to 8.4kb.

Stef
Very interested
Posts: 3131
Joined: Thu Nov 30, 2006 9:46 pm
Location: France - Sevres
Contact:

Re: zlib status

Post by Stef » Sat Dec 19, 2015 9:08 pm

Definitely interesting, it would be nice to lower ram usage as much as possible so we could use free space to store unpacked data :-) Is the unpacking implementation heavy or not (i mean in code size) ?

cero
Very interested
Posts: 338
Joined: Mon Nov 30, 2015 1:55 pm

Re: zlib status

Post by cero » Sun Dec 20, 2015 9:05 am

It's a few hundred lines of C. On x86_64, the compiled binary size is 13kb.

edit: Built for the Genesis, it's 6.2kb.

cero
Very interested
Posts: 338
Joined: Mon Nov 30, 2015 1:55 pm

Re: zlib status

Post by cero » Sun Dec 20, 2015 10:43 am

I tuned the speed-memory tradeoff, now it takes ~4kb RAM but decompresses 1% slower.

Stef
Very interested
Posts: 3131
Joined: Thu Nov 30, 2006 9:46 pm
Location: France - Sevres
Contact:

Re: zlib status

Post by Stef » Sun Dec 20, 2015 11:59 am

That sounds really good =) I think i will redo compression stuff so i will only keep this miniz and probably another method for "live" unpacking.
I will also keep specific methods for tilemap and tile data compression if they really worth it :)
Thanks for pointing that out. I saw nanozip offers a very good compression rate and also good unpacking speed depending selected option. Still I believe nanozip uses advances compression methods out of 68000 scope.

r57shell
Very interested
Posts: 478
Joined: Sun Dec 23, 2012 1:30 pm
Location: Russia
Contact:

Re: zlib status

Post by r57shell » Sun Dec 20, 2015 1:32 pm

cero wrote:aplib is closed source, and therefore I won't use it no matter how good it is.
Only compressor is closed source. What exactly you don't like in this fact?
Image

cero
Very interested
Posts: 338
Joined: Mon Nov 30, 2015 1:55 pm

Re: zlib status

Post by cero » Sun Dec 20, 2015 7:56 pm

@r57shell

A matter of principle. Closed source on a computer has both ethical and practical concerns (security); it can do all sorts of things and you have no way of knowing. Nor can you tune it, fix bugs, use it in ways the author didn't envision.

So in this sense I'd be more comfortable running closed source sw on an airgapped console, without network access, than on a computer where it may access/delete/modify my data, call home, or do all sorts of undesirable things.

r57shell
Very interested
Posts: 478
Joined: Sun Dec 23, 2012 1:30 pm
Location: Russia
Contact:

Re: zlib status

Post by r57shell » Mon Dec 21, 2015 6:40 am

Virtual Machine
or you can write your own aplib packer, as I did :lol: . But my reason was to ensure that it does maximal possible packing within unpacker code. And it figured out that it's not.

Hmm actually you can try contact author of aPLib :)
Image

cero
Very interested
Posts: 338
Joined: Mon Nov 30, 2015 1:55 pm

Re: zlib status

Post by cero » Mon Dec 21, 2015 9:13 am

Yes, a VM would provide separation, but that's an incredible PITA for something like this, that I don't want to use in the first place.

Post Reply