New XGM driver !

Stef · Post by **Stef** » Sun Nov 23, 2014 8:48 pm

I finally released a new version of SGDK including the new XGM driver =)
SGDK download page

It took sometime but here it is :
- 5 FM + 4 PSG + 4 PCM @14 Khz playback
- 16 priority levels for SFX play through PCM.
- support music pause / resume command.
It's still buggy on real hardware, it was working at some point but i broke it while doing later changes (there is still note playing after a pause command).
- no size limit for XGM music data or PCM data.
- PCM have their size and address aligned to 256 bytes.
- 100% running on Z80
- protection against DMA contention to preserve good PCM playback quality (not 100% guaranteed depending the driver load).

The XGM format is similar to VGM in the way it uses chip writes logging but is optimized for the Sega Megadrive where VGM support various systems. Because of that the XGM file size is usually about 30% smaller than VGM file (of course i am speaking about optimized VGM).
The advantage of using that type of format is that SGDK enable VGM playback directly through the XGM driver !
I developed a new tool (xgmtool) which convert a VGM file into a XGM file, it was a pain in the ass to convert my java tool in C but it's working now

Actually xgmtool can do a lot more than that, it is a powerful tool to optimize the Megadrive VGM file (require 1.5 vgm file at least) ! I was really disappointed with the current available VGM utilities... not able to find a working VGM optimizer, so i had to develop mine :-/
There are restrictions in the VGM optimization and the XGM convertion: the XGM format uses frame based timing so if a VGM file use sub frame timing then the conversion may fails...

Currently there is unfortunately no tracker taking advantages of the XGM 4 PCM channels but i hope it will in future (I will try to discuss about it with Delek and it would be nice to have it supported in the YMDj tracker as well) ^^
At least in the meantime it can be used to play multiples SFX at the same time !

For those who are interested, here are the XGM format specifications:
https://sgdk.googlecode.com/svn/trunk/bin/xgm.txt

Note that the Z80 XGM driver does not directly play the XGM file, it requires a compiled XGM file (thanks to xgmtool) which is very similar to the XGM format with some slight changes to make file parsing a bit easier for the Z80 =)

You can find an example in the usual sound sample:
source and binary

tryphon · Post by **tryphon** » Sun Nov 23, 2014 9:58 pm

Sounds amazing! (dans tous les sens du terme)

djcouchycouch · Post by **djcouchycouch** » Sun Nov 23, 2014 10:10 pm

I'm so happy I could cry. I've been waiting for something like this for years.

Great job!

kubilus1 · Post by **kubilus1** » Sun Nov 23, 2014 11:29 pm

Fantastic, can't wait to give this a whirl!

kubilus1 · Post by **kubilus1** » Mon Nov 24, 2014 2:59 am

Okay, I've played around a bit with the XGM driver mostly in the sample/sound program. It's pretty awesome! Here's a few things I noticed, however.

When playing a PCM sample, once a sample starts you can't play the same one again until it completes. For sound effects, it may be best to either restart the sample or play on another PCM channel.

There appears to be a slight delay when pressing a button to play a sample and the actual sample starting. I assume this is due to having to buffer the sample before playback. Would it be possible to pre-buffer the first however many bytes of samples?

Also, when playing the midnight resistance XGM, PCM samples won't play. Stopping the XGM, samples still won't play. After switching songs, samples appear to work again.

Great work though, this will be highly useful!

kubilus1 · Post by **kubilus1** » Mon Nov 24, 2014 3:10 am

Trying some other songs. I believe the PCM samples not being played with the midnight resistance XGM is due to that VGM using channel six to play music. The channel will need to be switched back and forth between music playback and sample playback, or the VGM to XGM conversion will need to deal with that.

I also confirmed this with some toejam slowjam which starts out using channel 6 for FM playback and then switches to samples at the end. At the beginning no PCM playback, once the track starts playing samples, PCM works again.

djcouchycouch · Post by **djcouchycouch** » Mon Nov 24, 2014 3:12 am

Stef, do you recommend a tracker for best support? Or are you using already written music and effects for your tests?

kubilus1 · Post by **kubilus1** » Mon Nov 24, 2014 3:17 am

This may just be a restriction of the XGM format, but VGM files that have samples in them don't appear to play correctly. A good example is the Sonic1 VGM in the sound sample program.

Stef · Post by **Stef** » Mon Nov 24, 2014 10:45 am

Thanks for the tests and the feedback kubilus1 !

When playing a PCM sample, once a sample starts you can't play the same one again until it completes. For sound effects, it may be best to either restart the sample or play on another PCM channel.

Actually that is just the implementation of the sound test sample (which switch between "start play" and "stop play" command). When you play a SFX, if the priority is equal or above the current played sample then it interrupts current sample and play the new one

At least that should be the case !

There appears to be a slight delay when pressing a button to play a sample and the actual sample starting. I assume this is due to having to buffer the sample before playback. Would it be possible to pre-buffer the first however many bytes of samples?

Indeed, that is a big limitation of the XGM driver, any PCM play command is delayed by ~3 frames due to the buffering. In the XGM music the PCM commands are moved 3 frames ahead (done by the XGM compilation process with xgmtool) so timing is almost ok but for SFX there is the delay.

I also confirmed this with some toejam slowjam which starts out using channel 6 for FM playback and then switches to samples at the end. At the beginning no PCM playback, once the track starts playing samples, PCM works again.

Indeed currently playing a PCM doesn't change the FM6 channel setting as the music can actually use it. If music set FM6 to FM then you can't play PCM anymore until the music set it back to DAC. Maybe that is something to change...

This may just be a restriction of the XGM format, but VGM files that have samples in them don't appear to play correctly. A good example is the Sonic1 VGM in the sound sample program.

Sample should be supported but it also depends from the input VGM format (< 1.5 not supported). I believe the included Sonic 1 VGM music does not have any sample in it, does it have ?
Also i know my XGM to VGM converter is *far* from perfect so it may have bug resulting in missing samples with some VGM file.

Stef, do you recommend a tracker for best support? Or are you using already written music and effects for your tests?

Unfortunately there is no tracker supporting it right now but i hope it will be the case of future =)
The only test i have currently is the sound sample rom, not really exciting i know :-/

kubilus1 · Post by **kubilus1** » Mon Nov 24, 2014 6:00 pm

Stef wrote: Actually that is just the implementation of the sound test sample (which switch between "start play" and "stop play" command). When you play a SFX, if the priority is equal or above the current played sample then it interrupts current sample and play the new one At least that should be the case !

Yes you are right! Just changing the logic to remove the else in src/main.c makes the sample play as many times as I like.

Indeed, that is a big limitation of the XGM driver, any PCM play command is delayed by ~3 frames due to the buffering. In the XGM music the PCM commands are moved 3 frames ahead (done by the XGM compilation process with xgmtool) so timing is almost ok but for SFX there is the delay.

There might not be much to do about this, and it really may not be a big enough delay to be a big deal. I was thinking perhaps though, that maybe sound effects could be first 'registered' with the driver so that the beginning part is already buffered. This would depend on how much memory is still available in z80 space.

Indeed currently playing a PCM doesn't change the FM6 channel setting as the music can actually use it. If music set FM6 to FM then you can't play PCM anymore until the music set it back to DAC. Maybe that is something to change...

This is something I had to deal with in the VGM driver. I basically would track the state of channel 6. If need be I would switch it from FM to PCM, play the sample (or part of the sample usually) and return the state to the previous state when done. Essentially I would interleave PCM and FM on channel six with a priority for PCM sound.

Sample should be supported but it also depends from the input VGM format (< 1.5 not supported). I believe the included Sonic 1 VGM music does not have any sample in it, does it have ?

The Sonic1 should be a v 1.6+ VGM file (actually exported from Deflemask). It contains a couple samples for the drums and they are in the newer VGM PCM sample format (indexed from the start of the VGM file). In the XGM driver, these will play, sort of, but sound really bad like they are not starting at the right point in the sample or something.

Also there was one VGM file that caused the xgmtool to segfault, I'll try to take a look why when I get home from work and have some time.

Davidian · Post by **Davidian** » Mon Nov 24, 2014 6:19 pm

Hi!

I'm doing some tests. For Antarex we are using Deflemask and the XGM driver do things right with FM but PCM doesn't work properly (the tracks for Antarex work perfectly with current SGDK VGM driver).

However, in my old tracks made with VGM Music Maker PCM seems to work perfectly.

If you need more help or info, please let me know

Stef · Post by **Stef** » Mon Nov 24, 2014 6:35 pm

Thanks for the report ! I guess there is a bug in how i am handling VGM 1.6+ version with xgmtool. I have to admit the 1.6 PCM command addition in the VGM format looks a bit... weird, at least i had some headache in understanding their logic :p So i may have made mistakes in handling some commands. I will try to test that asap (i am currently pretty busy with job :-/) !

neologix · Post by **neologix** » Mon Nov 24, 2014 10:13 pm

Re VGM programs - maybe get in contact with ValleyBell about it? I know he made an entire suite of command-line programs to manipulate VGMs, including an optimizer, and while I haven't touched it in a long time I myself previously started work on an updated version of VGMTool that would also be cross-platform.

Stef · Post by **Stef** » Mon Nov 24, 2014 10:30 pm

There might not be much to do about this, and it really may not be a big enough delay to be a big deal. I was thinking perhaps though, that maybe sound effects could be first 'registered' with the driver so that the beginning part is already buffered. This would depend on how much memory is still available in z80 space.

I can't really do that, what i can do however is to lower the buffer size but i'm afraid it can lead to garbled PCM when the XGM command parsing is high (the driver is designed to give priority on XGM parsing).

This is something I had to deal with in the VGM driver. I basically would track the state of channel 6. If need be I would switch it from FM to PCM, play the sample (or part of the sample usually) and return the state to the previous state when done. Essentially I would interleave PCM and FM on channel six with a priority for PCM sound.

Yeah, i think that is something i should do as well... the problem is that i don't have much free room for adding code logic, i eat almost the whole 8 KB Z80 memory, i need to optimize the code space.

Re VGM programs - maybe get in contact with ValleyBell about it? I know he made an entire suite of command-line programs to manipulate VGMs, including an optimizer, and while I haven't touched it in a long time I myself previously started work on an updated version of VGMTool that would also be cross-platform.

To be honest i tested ValleyBell tools and experienced some issues... I remember about a tool to optimize VGM in order to replace wait command in frame wait and I always ended by getting VGM with wrong timing playback (about 2x too slow), also the VGM optimizer never worked right for me :-/
It's a shame as it is definitely not a trivial task to optimize a VGM file and i spent many time in developing my own...

bastien · Post by **bastien** » Tue Nov 25, 2014 5:47 am

Finally you release it

Thanks you very much Stef and congrats for your works

I will test as soon as possible