Standart data formats in SMD games

Shiru · Post by **Shiru** » Thu Jan 01, 2009 12:49 pm

After MOD file in Toy Story I wondered if any other games using standart data formats and tried to check large set of games (not complete romset, though) with Fast Module Extractor. Here is results:

Alladin - Creative Voice File
Best of the Best - MIDI music
Blood Shot - two LBM pictures 320x200
Battletech - some data in IFF
Cal Ripken Jr. Baseball - Creative Voice File
Cool Spot - Creative Voice File
Davis Cup World Tour Tennis - MIDI music
Dazexmas - some data in IFF
Death and Return of Superman - Windows WAVE sounds
Dune - The Building of a Dynasty - some data in IFF
F-15 Strike Eagle II - 8SX sounds (8-bit SVX)
James Bond - The Duel - 8SX sounds
Jammit - Creative Voice File
Lemmings - some data in IFF
Mick & Mack as the Global Gladiators - Creative Voice File
Nightmare Circus - some data in IFF
Outlander - 8SX sounds
Power Drive - 8SX sounds
VR Troopers - some data in IFF
Waterworld - 8SX sounds
We're Back! - A Dinosaur's Tale - some data in IFF

Graz · Post by **Graz** » Sun Jan 04, 2009 1:57 pm

I used the same program to extract a large set of ROMs - almost complete. The extractor did find a lot of data, but a good chunk of it (~30%) isn't meaningful - perhaps the headers just matched the binary data by coincidence. There are some interesting things to note:
The 'data in IFF' stuff all has a FORM ID of SEGA TLHD and chunk names like CMAP and TILE. This looks like it could be a standard tile set format. Does anyone know what application produced that?
While WAV, and MID is not particularly difficult to parse, it does make more sense to do this offline and include 'ready to run' data in ROM. Certainly when I'm packing ROMable data, I'll strip out everything unnecessary, fixup offsets to absolute pointers and do anything else I can to make the data either smaller or easier to deal with at runtime. I wonder why developers decided to put unprocessed, standard files right into the ROM. For example, VOC, 8SX and WAV have little endian data in them - not good for 68K. I even found a couple of JPEGs (big ones) in there.

Shiru · Post by **Shiru** » Sun Jan 04, 2009 2:16 pm

Graz wrote:While WAV, and MID is not particularly difficult to parse, it does make more sense to do this offline and include 'ready to run' data in ROM.

I also have found few MIDI files in PC-Engine games, and one of those games is Davis Cup; and I found MIDI files in SNES and Gameboy versions of Best of the Best as well. So probably this related to multiplatform development.

Graz wrote:I even found a couple of JPEGs (big ones) in there.

This is the case of matched headers, not real JPEG for sure.

Huge · Post by **Huge** » Sun Jan 04, 2009 2:52 pm

Graz wrote:I wonder why developers decided to put unprocessed, standard files right into the ROM.

My guess is that so they can be accessed faster, eg. there is less (or no) burden put on the 68k/z80 by decoding the data, freeing up processing power for other tasks.

LocalH · Post by **LocalH** » Sun Jan 04, 2009 5:44 pm

Graz wrote:The 'data in IFF' stuff all has a FORM ID of SEGA TLHD and chunk names like CMAP and TILE. This looks like it could be a standard tile set format. Does anyone know what application produced that?

Ah, good old IFF. Takes me back to my Amiga days

IFF FORM/CHUNK registry

Concerning the SEGA.TLHD format, it seems like someone made some MD graphics tools for the Amiga and decided to use IFF as the framework. I wouldn't be surprised if the data is just otherwise normal MD data, although I haven't looked myself yet. Doesn't appear to be an "official" IFF format though.

ILBM and 8SVX are also IFF-based, so I'm not sure what endian issues you're referring to with 8SVX, since IFF is an Amiga-based format, and thus the processor matches.

Graz · Post by **Graz** » Sun Jan 04, 2009 7:48 pm

Shiru wrote:This is the case of matched headers, not real JPEG for sure.

They open just fine in Paintshop Pro. The image size is small, but the files are about 7megs(!). I have no idea what all the extra data is or where it came from (the ROMs are at most 4 megs), but they do appear to be valid JPEGs.

Huge wrote:
Graz wrote:I wonder why developers decided to put unprocessed, standard files right into the ROM.
My guess is that so they can be accessed faster, eg. there is less (or no) burden put on the 68k/z80 by decoding the data, freeing up processing power for other tasks.

Actually, including the standard file in the ROM requires that the 68K or Z80 parses the data. Be preprocessing it during the build, it is possible to include data that can be accessed directly by the hardware. Who cares about chunk IDs and endianness issues when you have known data in ROM at a known address?

LocalH wrote: ILBM and 8SVX are also IFF-based, so I'm not sure what endian issues you're referring to with 8SVX, since IFF is an Amiga-based format, and thus the processor matches.

My mistake on the 8SVX format. It's been a while since I've seen any 8SVX files.

Shiru · Post by **Shiru** » Sun Jan 04, 2009 8:09 pm

Graz wrote:They open just fine in Paintshop Pro. The image size is small, but the files are about 7megs(!). I have no idea what all the extra data is or where it came from (the ROMs are at most 4 megs), but they do appear to be valid JPEGs.

I'm sure this is sort of glitch. JPEG standart was defined in 1992-94 and got popularity some later; decoding will take quite much time on SMD (up to minutes); it makes no sense to use lossy compression on low-res images with palette (compression artefacts will be very noticeable).

Huge · Post by **Huge** » Sun Jan 04, 2009 9:00 pm

Graz wrote:Actually, including the standard file in the ROM requires that the 68K or Z80 parses the data. Be preprocessing it during the build, it is possible to include data that can be accessed directly by the hardware. Who cares about chunk IDs and endianness issues when you have known data in ROM at a known address?

That is true, but I was thinking about data compression to reduce the size.

TmEE co.(TM) · Post by **TmEE co.(TM)** » Mon Jan 05, 2009 1:15 am

For example when you use ALIGN in your assembler, you get empty data in your ROM, but maybe some assemblers just did a file size increase to what is needed, and that will leave anything you had on your HDD into the newly added part... make a 1 byte file and increase its size to 10MBytes using a DOS int, you will most definitely see some junk...
One beta/alpha of Sonic2 has quite a bit of raw assembly in it, and this padding stuff is most likely the cause for them to be there.
I guess same would go for the JPG files, and other stuff not used in a game.

BTW, there is a multi platform tool called The Universal Map Editor, I know on MD the Global Gladiators use it... the demo I downloaded from Programmersheaven even had some graphics from this game in the example

Oerg866 · Post by **Oerg866** » Mon Jan 05, 2009 3:07 am

Yeah, so,

Nineko saw this thread and then has researched some awesome stuff regarding that davis tennis game and MIDI.

http://forums.sonicretro.org/index.php?showtopic=13488

tomaitheous · Post by **tomaitheous** » Mon Jan 05, 2009 6:42 am

I even found a couple of JPEGs (big ones) in there.

ROMs can contain left over files and data from the original development stage. Stuff that gets accidentally included into the rom for the unused areas. I've seen this a lot in PC-Engine ROMs. Other game data(even from other systems), source, development text, etc. Zelda for GBC is a good example of source accidentally including into the master image. Doing a text search with 0x00(ascii-0x20) offset yields lots of hidden text too

Hell, there's even a game with source for the whole project. I won't mention the name of the game for obvious reasons.