Compression and SGDK

SGDK only sub forum

Moderator: Stef

Post Reply
themrcul
Very interested
Posts: 116
Joined: Fri Apr 15, 2016 2:21 pm

Compression and SGDK

Post by themrcul » Sat Jun 17, 2017 12:06 pm

Hi Stef and others,

I am trying to get compression to work with my engine.

When I don't use compression with rescomp, everything builds fine.

However, there are two separate problems with compression when I try to use it.

Firstly, when I use the option for fast compression (2 / FAST / LZ4W), rescomp does not export the tileset with compression. The .s file is the same size as no compression, and the compression flag in the .s file is set to 0.
VDP_loadTileSet still works, because it is not compressed.

Secondly, when I use slow compression (1 / SLOW / APLIB), it does compress the tileset, and the resultant .s file is half the size. This is awesome!
However it will not load in game.

I have tried just using VDP_loadTileSet, but it returns 0, which is fail to decompress.
When I allocate a TileSet in memory first with MEM_alloc(), and even when I use a lot more memory than it should need, unpackTileSet returns 0 for failure.

Firstly, do you know of fast compression not working with rescomp?
Secondly, how can I get slow (aplib) compression to work with tilesets? The tileset I am trying to decompress is 264 tiles in size, which should be 8448 bytes (should fit well within memory). The source is a 4bit BMP file. I am trying to load them to the VDP on map loading, so VDP is disabled at the time of usage. Toggling DMA usage on/off does not change the result (still fails).

Thanks for your time and help!

cero
Very interested
Posts: 338
Joined: Mon Nov 30, 2015 1:55 pm

Re: Compression and SGDK

Post by cero » Sat Jun 17, 2017 6:13 pm

For the LZW4 compression to work, you need Java installed.

Stef
Very interested
Posts: 3131
Joined: Thu Nov 30, 2006 9:46 pm
Location: France - Sevres
Contact:

Re: Compression and SGDK

Post by Stef » Sat Jun 17, 2017 10:09 pm

As cero said fast compression (LZ4W) need java as the compressor is wrote in java.
For APLIB compression, if unpacking fails there is great luck there is not enough memory available to unpack the tileset. Are you trying to load a huge tileset ??

themrcul
Very interested
Posts: 116
Joined: Fri Apr 15, 2016 2:21 pm

Re: Compression and SGDK

Post by themrcul » Sat Jun 17, 2017 11:11 pm

Hi guys, thanks for your help so far.

Hmmmm I should have Java installed. Is there a particular command it is using that I can test to see if my system is configured correctly? Or is it just Java.exe? Does it need to be 32bit Java vs 64bit?

With the tileset, it is 264 tiles. So 264 x 32 bytes is 8448 bytes. I have 50kb free in RAM, so that should be plenty. Is there something I'm missing with that?

themrcul
Very interested
Posts: 116
Joined: Fri Apr 15, 2016 2:21 pm

Re: Compression and SGDK

Post by themrcul » Sat Jun 17, 2017 11:38 pm

Ok testing manually, rescomp says it can't find jarfile lz4w.jar.
It exists in the SGDK\BIN directory, but I was calling it from another directory.

When I called rescomp from the bin directory, it compressed successfully.
So, no compression = 40kb .s file
Slow = 20kb .s file
Fast = 29kb .s file.

The compression at least is all working for me now!

However when I ran my game with the map tiles using LZ4W compression, VDP_loadTileSet() returned 0 and did not upload the tileset to VDP.

When I call:

Code: Select all

TileSet* ts = (TileSet*)MEM_alloc(32000);
if(unpackTileSet(tileset->tileData, ts))
	result = VDP_loadTileSet(ts, tileset->vramDestination, 1);
else
	Debug_PrintText("Failed to unpack");
Or

Code: Select all

TileSet* ts = unpackTileSet(tileset->tileData, 0);
if (ts)
	result = VDP_loadTileSet(ts, tileset->vramDestination, 1);
else
	Debug_PrintText("Failed to unpack");
The variable ts becomes 0 which signifies a failure to decompress.
(I know that 32kb mem allocation is overkill, I have tried a few sized allocation from 8448 bytes, 16kb to 32kb just in case but all fail).

So for the tiles I am using, both fast and slow decompression is not working. Is there any way I can debug the reason why decompression is failing? Surely 264 tiles is not too much to decompress, particularly when the VDP is disabled, is it not?

Stef
Very interested
Posts: 3131
Joined: Thu Nov 30, 2006 9:46 pm
Location: France - Sevres
Contact:

Re: Compression and SGDK

Post by Stef » Sun Jun 18, 2017 8:23 pm

It' weird that it require lz4w.jar file to be located somewhere else, normally rescomp is always called from its 'bin' folder (if you are using the SGDK makefile) and so it tries to call lz4w.jar from the same folder.

About the code you're using, you have to pass the tileset pointer to unpackTileset(..) method, not the tile data itself:

Code: Select all

TileSet* ts = unpackTileSet(tileset, NULL));
Using NULL as second argument will let SGDK to handle memory allocation itself.

themrcul
Very interested
Posts: 116
Joined: Fri Apr 15, 2016 2:21 pm

Re: Compression and SGDK

Post by themrcul » Mon Jun 19, 2017 10:16 pm

Hi Stef,
Thanks for your reply and sorry for creating confusion, I am using a custom build process, which is calling rescomp from a c# level editor I am making. However since I didn't set the starting directory for the rescomp command window process, it was starting in C:\Windows\System32 and therefore couldn't find lz4w.jar.
Once I set my build process to start in the %GDK%\bin directory, FAST compression now works.

--
For decompression failing, I also created confusion by leaving out my structure definition. When I am calling tileset->tileData, I am pointing towards the TileSet* structure. tileset is a MDPMTileset* variable with the following structure definition:

Code: Select all

typedef struct MDPMTileset
{
	const TileSet* tileData;
	u16 numTiles;
	u16 vramDestination;
}MDPMTileset;
So in my code tileData is actually a const TileSet*.
But calling TileSet* ts = unpackTileSet(tileset->tileData, NULL)); still fails to decompress...

Stef
Very interested
Posts: 3131
Joined: Thu Nov 30, 2006 9:46 pm
Location: France - Sevres
Contact:

Re: Compression and SGDK

Post by Stef » Tue Jun 20, 2017 8:32 am

What you can do eventually is to recompile the library in debug (debug target in makefile) so it will generate logs in the Gens KMog output window, that may help to understand why you compressed data fail to unpack.

themrcul
Very interested
Posts: 116
Joined: Fri Apr 15, 2016 2:21 pm

Re: Compression and SGDK

Post by themrcul » Tue Jun 20, 2017 8:35 am

Great, thanks Stef, I'll try that out.

themrcul
Very interested
Posts: 116
Joined: Fri Apr 15, 2016 2:21 pm

Re: Compression and SGDK

Post by themrcul » Sat Jun 24, 2017 9:41 am

Hi Stef,
Thanks for your help with this.
I have just upgraded the SDK to the latest version you have just released with the new GCC!

The problem was indeed I was out of memory.

I didn't think it could be possible at first, since my project is so simple.

However I dug deeper, seeing that I was only allocating ~7KB for my sprite engine, and that was the only thing I was "allocating".
So the rest had to be objects I've generated in my program that weren't const.

That was where all the memory was going. After I consted everything exported, BAM, decompression worked.

Thanks for your help Stef, it pointed me in the right direction, and helped me fix a few bugs in my engine along the way!

Stef
Very interested
Posts: 3131
Joined: Thu Nov 30, 2006 9:46 pm
Location: France - Sevres
Contact:

Re: Compression and SGDK

Post by Stef » Sat Jun 24, 2017 2:19 pm

Really glad you fixed your problem :)
Indeed you really need to declare all your constant table as const otherwise they will be put in main memory.

Post Reply