Graphics Compression Formats

Ti_ · Post by **Ti_** » Tue Jun 18, 2013 2:54 pm

sega16 wrote:You could check out https://github.com/sikthehedgehog/mdtoo ... aster/uftc
it is made for doing decompressing sprites.
Also he has another compression called slz that is more of a generic compression
https://github.com/sikthehedgehog/mdtoo ... master/slz

Someone can make windoz binaries for packers tools?

Ti_ · Post by **Ti_** » Tue Jun 18, 2013 7:27 pm

I've added a new version of ROM with Nemesis. (same link)

Compression actually better than LZSS, but not always: some picture comperessed bad - greater on 6kb, on others size lower on 5kb.
Speed: more slowly than lzs.
Ram to usage: only 504 bytes needed!

On average size of all pics "nemesis" better than lzss.
So, summary it better. But designed for compress only tiles.

Ti_ · Post by **Ti_** » Wed Jun 19, 2013 8:55 am

Added Packfire 'Tiny' version ROM.

About 1kb/picture better compression than aplib.

pluses: best compression rating, except packfire large.
minuses: very slow decompression. (possible make not so slow, if edit code)
Compress max 32,768kb file. (I cut my files on 2 parts to get it work).

Useful: as alternate to large version - faster and not needs extra ram.
as alternate to aplib - slower, but can save extra space.

r57shell · Post by **r57shell** » Wed Jun 19, 2013 2:22 pm

Heya! Guys. I just tried myself with making packers. :]
Everyone who want to test my packer, here it is:
http://elektropage.ru/r57shell/elektro_pack.zip
It is realy slow packing. I will optimize some things later.

Edit: Packer updated:

End marker bug fixed. Packer did not specify the end of archive.
Impemented complicated optimization. Now packing is much faster.
Speed setting added.

Unpacking needs optimizations too.

Ti_ · Post by **Ti_** » Thu Jun 20, 2013 8:09 am

r57shell wrote:Heya! Guys. I just tried myself with making packers. :]
Everyone who want to test my packer, here it is:

Okay. I've test it, your version added to test rom, archive updated.

Packing - updated version much better. But having trouble with packing some pictures with many empty space. (previous version may pack over 1hour, new version 20minutes). [last picture with minotaur]
Second picture packed enough fast. (maybe 30sec-1min).

Compression rating: close to aplib. (~averate ~500 bytes more bad than aplib)
DeCompression speed: slower than others, but around 2 seconds to unpack. (not as awful packfire 3-20 seconds)

If it's just a try to "create packer" - that good a try. I think it compression rating better than RNC.'
But overall not better than others, and don't have unique things. (I've tested only on tiles, not other data)

Wanna someone try to launch Packfire Rom on hardware to see actual value (all emulators shows different) ?

r57shell · Post by **r57shell** » Thu Jun 20, 2013 11:24 am

Updated archive, fast unpacker on m68k side added.
I know how to make it little bit more faster. But, then I need to swap some bits in format. I think, I will do it, later...

Hehehe. I made some tests with LZMA from LZMA SDK (used in 7z).
31552 bytes packed into 13203 with header. Header can be omitted. Header size = 12 bytes. For example, my packer on this file gives 15895 output.
But, as expected... very slow unpacking (about 10 sec) + memory usage (about 500 bytes from stack) + big code (about $1F00).
But, it was fun.
It can be very good unpacker for Genesis, but, needs some time to optimize under this platform.

r57shell · Post by **r57shell** » Thu Jun 20, 2013 12:30 pm

sega16 wrote:You could check out https://github.com/sikthehedgehog/mdtoo ... aster/uftc
it is made for doing decompressing sprites.
Also he has another compression called slz that is more of a generic compression
https://github.com/sikthehedgehog/mdtoo ... master/slz

1) no compression, just duplicate cleaning.
2) compression, fast and low.
For me, nothing interesting

.

Ti_ · Post by **Ti_** » Thu Jun 20, 2013 1:22 pm

r57shell wrote: Hehehe. I made some tests with LZMA from LZMA SDK (used in 7z).
31552 bytes packed into 13203 with header. Header can be omitted. Header size = 12 bytes. For example, my packer on this file gives 15895 output.
But, as expected... very slow unpacking (about 10 sec) + memory usage (about 500 bytes from stack) + big code (about $1F00).

Packfire is remake of 7zip too, right? (but without $2kb code)

I update my archive:
added Elektro more faster decompress version. (~2x times faster as i see).

Added Thunder force III RLE compression: what can't say - It's not useful as 'fast'. 2 times slower than lzss.
Only one good thing: rle doesn't needs any ram at all.
Also dithered pictures compressed size more than uncompressed.

Ti_ · Post by **Ti_** » Thu Jun 20, 2013 9:28 pm

Ti_ wrote: Packfire is remake of 7zip too, right? (but without $2kb code)

Well, I've downloaded LZMA sdk too.

I packed my data:

lzma e apelsin.smd apelsin.pck -d16 -fb220 -mc256 -lc0 -pb0 -mfbt2

cut off header, (keeps only first 4 bytes that unpacked size).
Total file size 7107 bytes vs 7136 packfire.

Replaced old file with new.. and...
it's unpacked !. (1 frame longer. hah)

So, Packfire it's a just LZMA.exe with cutted header, and some settings and other 'tiny method'.

Ti_ · Post by **Ti_** » Fri Jun 21, 2013 8:04 am

Test ROM builder (all included):

LZMA_SEGA.7z

Adjusting settings will not help improve decode speed. But you can compress more powerful than packfire (on 10-20 bytes huh).
Some settings can't be changed (decoder locked to it).

"-fb220 -mc256" gives better result for me, than max possible settings -fb273 -mc9999, so I kept it.

*.lzma can be easily opened with 7-zip

Ti_ · Post by **Ti_** » Fri Jun 21, 2013 3:34 pm

r57shell wrote: It can be very good unpacker for Genesis, but, needs some time to optimize under this platform.

I've readed about LZHAM - new algorithm, it's compress not so powerful as LZMA, but decompress 3x times faster.
Maybe it can be used on Sega too?

Code: Select all

lzham_alpha7_r1\bin>lzhamtest_x86
 -x -e c apelsin.smd apelsin.lzham

Input file size: 38400, Compressed file size: 7858, Ratio: 79.54%

http://code.google.com/p/lzham/wiki/Com ... niquesUsed

r57shell · Post by **r57shell** » Fri Jun 21, 2013 4:53 pm

Ti_ wrote: I've readed about LZHAM - new algorithm, it's compress not so powerful as LZMA, but decompress 3x times faster.
Maybe it can be used on Sega too?

First, please fix quote several posts up.
Second: I looked in the code of LZHAM. I don't like such implementation.
Very big code, with many branches, It's called Spaghetti code. :S
And many of source files.
For example: LZMA Decompression in SDK: only one source with one header.

Ti_ · Post by **Ti_** » Sun Jun 23, 2013 7:39 am

Added lzss koei (LZB). Used in Genghis Khan II and other koei games.

Sega_Pics_packed.7z

compression rating: very good.
memory usage: no usage (reads from vram)
speed: slow - 3-4 sec. (actually in ROM exists 2 versions - second to unpack to ram - I've not tested it's speed)

r57shell · Post by **r57shell** » Mon Jun 24, 2013 10:34 pm

Ti_ wrote: "-fb220 -mc256" gives better result for me, than max possible settings -fb273 -mc9999, so I kept it.

For me, "best" is:

Code: Select all

-fb273 -lc0 -lp0 -pb0 -mfhc4 -mc100000

Repack all with some "best" settings.

What aplib unpack code do you use in demo?

Ti_ · Post by **Ti_** » Tue Jun 25, 2013 6:38 am

Code: Select all

-fb273 -lc0 -lp0 -pb0 -mfhc4 -mc100000

Every file can give better results on different settings. And it's not always max, but in ~50 bytes range.
For me first file gives best at -fb231 and second with -fb227. But -fb 228 and 226 gives bad result.

Some kind of random, depends on file.
Also I've tested real memory usage ("probs buffer") is not 15980 bytes; for 38400 files it's always 5228 bytes; for others not tested.

r57shell wrote:
Ti_ wrote: "-fb220 -mc256" gives better result for me, than max possible settings -fb273 -mc9999, so I kept it.
For me, "best" is:
Code: Select all
-fb273 -lc0 -lp0 -pb0 -mfhc4 -mc100000
Repack all with some "best" settings.

What aplib unpack code do you use in demo?

Code I modify to use macros instead of bsr
viewtopic.php?t=703&highlight=aplib

, plus slightly some other changes, I don't remember (maybe subq->dbf?); this speed up unpacking twice:

in demo:
aplib_decrunch.txt
2nd for max pack size (?32k):
aplib_32000.txt

LZMH - new algorithm, 3times faster than 7z; as I understand exists only dll in 7-zip version 4.56 alpha:
LZMH
possibly to write unpack code?