New XGM driver !

[gasega68k]

Stef: I've been having some problems in converting some files from "VGM" to "XGM" I downloaded several VGM files (from project2612.org) of Sonic 3 and Sonic 3D Blast, and wanted to use them in the "sonic kart" but when I try to convert them to XGM format with xgmtool, it displays many "warnings" like this: "warning:. more than one PCM command in a single frame," and when I test the "XGC" files with the XGM driver simply do not work (no sound).

Also try the VGM logging of Kega Fusion and then converting it, but it's the same, nothing.

I also did the test with the SMPSplay (is a utility to play SMPS files, I got in sonicretro, and has support for logging VGM (v1.60)) but in this case after convert to XGM, then music has no PCM samples, only fm and PSG. I also tried the gensGs-r7 emulator that has VGM logging (I think v1.5), but it is the same after converting only FM and PSG.


The XGM driver I'm using is a small variation of the code you posted in Basiegaxorz forums (in ASM) using debugger I realize that when it enters the routine to play music ("xgm_startPlayMusic"), it tries to load more than 64 samples (sample table id), I tried other vgms that work well and only load a few. I tried the intro music of g-zero I did, which only has one sample, effectively only loads one.

If I only export a little time with Kega fusion rather than the entire track (in this case I tried five seconds of "Green Grove" of Sonic 3D Blast) works well, but still "loaded" many samples, the problem may be there .

Do not know if this will be a problem of the xgmtool, or has something to do with the games "Sonic 3" or 3D Blast (tested with sonic 1 and 2 and work well).

In the rar file I'm including several examples that went wrong, also the XGM driver (I also included a readme.txt to explain each file).


http://www.mediafire.com/download/nou2w ... _files.rar

[Stef]

Thanks for the report gasega68k, i am actually looking for problematic VGM files I do know that i will always found some problems depending how the VGM files has been dumped and the XGM conversion can be painful, still i try to support almost all them so it is always nice when i have a new example which does not work. Normally a song should not contains more than 64 samples so here the problem is about PCM sample recognition... I will check your VGM files asap and work out the xgmtool to fix the conversion issues. Something to know is that xgmtool requires at least VGM in version 1.5, it just does not support older VGM format (well actually i can add support for it but without PCM which is not really interesting).

By the way i updated the XGM driver posted in the DevSter (Basic) forum, actually it uses a newer version that the one available in the last SGDK (better handling of the main bus contention).

Something to know about the XGM driver as i saw you got some issues on real hardware. The XGM driver uses about 75% of Z80 CPU time in the PCM 4 channels mixing process and for that it requires to access the main bus. It also requires the main bus to prefetch the XGM music data and parse the PSG command (PSG is connected to main BUS) so if your game uses DMA for more than 20% / 25% of the frame time it may impact the speed of the XGM driver and you may experience slowdown in music playback. I will try to improve that part so at least the FM/PSG music data keep the good timing but PCM will be impacted as there is no way to reduce the PCM mixing process time.

To summarize: the XGM driver has been developed for game using DMA only during VBlank area and it may not work correctly when a game extends the blanking area to do larger DMA (as this can stop the Z80 execution for too long time).

[gasega68k]

Edit by Stef: Sorry gasega68k, i used the 'edit' button instead of 'reply' (as i am moderator in this section) and so lost a part of your original message !

I forgot to mention that when I used SMPSplay to log VGM, and then I used xgmtool to optimize the resulting VGM had no samples (I use winamp to play VGM).
Below I leave a RAR with both files (one made with SMPSplay and the other optimized with xgmtool). By the way the music extracted with SMPSplay has a better quality samples.

...lost part...

And on the DMA, these demos (g-zero and sonickart), to transfer the buffer to the VRAM, about 37 lines (6144 bytes / 166) are required, although this transfer occurs every two frames (30 fps) instead of every frame, but I will add the DMA for sprites (SAT), and also have planned to make scaling (in some sprites) and that requires another buffer to transfer with dma.
Another thing, during the scaling of the logo of G-zero, there is a greater transfer of dma, exactly 16KB, which require about a 99 lines, but also occurs at 30 fps, but apparently not affect the music, or maybe because the music is less "complicated".

...lost part...

Re-edited to include again the RAR file:
http://www.mediafire.com/download/60429 ... gm_vgm.rar

[Stef]

I forgot to mention that when I used SMPSplay to log VGM, and then I used xgmtool to optimize the resulting VGM had no samples (I use winamp to play VGM).
Below I leave a RAR with both files (one made with SMPSplay and the other optimized with xgmtool). By the way the music extracted with SMPSplay has a better quality samples.

I will check that, maybe the VGM export version is not supported. What is the input of SMPSPlay ? i guess it uses SMPS driver music format right ? where you can find that type of rip ?
Also about the sample quality, XGM force to resample to 14 Khz (as the driver play at fixed 14 Khz rate) so if the original samples were at higher rate you loss in quality. Actually even if the samples were at a lower rate, because of the very simple conversion applied (linear interpolation) it will always sounds worse in the XGM driver :-/
I really need to improve the resampling algorithm... One solution is to resample samples at 14 Khz with another tools before doing the XGM conversion but that is not really possible when you work from VGM files.

And on the DMA, these demos (g-zero and sonickart), to transfer the buffer to the VRAM, about 37 lines (6144 bytes / 166) are required, although this transfer occurs every two frames (30 fps) instead of every frame, but I will add the DMA for sprites (SAT), and also have planned to make scaling (in some sprites) and that requires another buffer to transfer with dma.
Another thing, during the scaling of the logo of G-zero, there is a greater transfer of dma, exactly 16KB, which require about a 99 lines, but also occurs at 30 fps, but apparently not affect the music, or maybe because the music is less "complicated".

Well that should fit in your Z80 time budget
The XGM driver buffer is about ~3.5 frames so even if you use more than 20% of a single frame with a huge DMA (as the intro scaled logo), as soon you don't cumulate more than 20% on 3.5 consecutive frames it's ok. In your case the driver will use more Z80 CPU time on odd frame to stay in sync.

Something to know, you can use this method:
https://github.com/Stephane-D/SGDK/blob ... und.h#L597
to help the XGM driver not accessing the 68K BUS during DMA .

[gasega68k]

I will check that, maybe the VGM export version is not supported. What is the input of SMPSPlay ? i guess it uses SMPS driver music format right ? where you can find that type of rip ?
Also about the sample quality, XGM force to resample to 14 Khz (as the driver play at fixed 14 Khz rate) so if the original samples were at higher rate you loss in quality. Actually even if the samples were at a lower rate, because of the very simple conversion applied (linear interpolation) it will always sounds worse in the XGM driver :-/
I really need to improve the resampling algorithm... One solution is to resample samples at 14 Khz with another tools before doing the XGM conversion but that is not really possible when you work from VGM files.

Sorry, but I think I did not explain well , when I talked about the quality of the samples I meant that the quality of the samples in the VGMs created with SMPSplay is better than you get with emulators, I think it has to do as how SMPSplay plays the files (without interruption of z80 due to DMA).

I found the "SMPSplay" by a post made by "ValleyBell" in sonicretro forum, there is the SMPSplay (SMPSplay 2.01) and also is the "SMPS Research Pack v3" containing a lot of rips of several games (SMPS files to reproduce with SMPSplay), as well as several SMPS drivers, and also disassembled SMPS drivers, is very interesting. :
http://forums.sonicretro.org/index.php? ... 32473&st=0

Well that should fit in your Z80 time budget
The XGM driver buffer is about ~3.5 frames so even if you use more than 20% of a single frame with a huge DMA (as the intro scaled logo), as soon you don't cumulate more than 20% on 3.5 consecutive frames it's ok. In your case the driver will use more Z80 CPU time on odd frame to stay in sync.

Ok, so if my calculations are correct, I believe that in the intro of the G-Zero logo must be within the limits, something close to 19% (20% of a frame is 52.4 lines, then 99 lines of DMA / 2 = 49.5 lines per frame, this means that in the frame that is not doing the DMA, the Z80 will be using almost 100% of the time).

Something to know, you can use this method:
https://github.com/Stephane-D/SGDK/blob ... und.h#L597
to help the XGM driver not accessing the 68K BUS during DMA .

Well I had not seen this:

Code: Select all

void SND_set68KBUSProtection_XGM(u8 value);

definitely I have to use this method to be more secure.

[Stef]

Sorry, but I think I did not explain well , when I talked about the quality of the samples I meant that the quality of the samples in the VGMs created with SMPSplay is better than you get with emulators, I think it has to do as how SMPSplay plays the files (without interruption of z80 due to DMA).

Oh ok, indeed sample playback in emulator generally sucks, you really have to test it on real hardware to actually know how it gets
Good to know that playing music from the SMPSplay preserv the PCM quality at least !

I found the "SMPSplay" by a post made by "ValleyBell" in sonicretro forum, there is the SMPSplay (SMPSplay 2.01) and also is the "SMPS Research Pack v3" containing a lot of rips of several games (SMPS files to reproduce with SMPSplay), as well as several SMPS drivers, and also disassembled SMPS drivers, is very interesting. :
http://forums.sonicretro.org/index.php? ... 32473&st=0

Thanks for the link, i did know about the player, but not about the music resources =)

Ok, so if my calculations are correct, I believe that in the intro of the G-Zero logo must be within the limits, something close to 19% (20% of a frame is 52.4 lines, then 99 lines of DMA / 2 = 49.5 lines per frame, this means that in the frame that is not doing the DMA, the Z80 will be using almost 100% of the time).

Yep, it means that the Z80 should be about 100% busy during the available time in this section but that is not a problem after all, 68000 is all free

By the way, i got some troubles in fixing my xgmtool, i made progress and so got some of your musics are now correctly translated to XGM but then a lot of others were broken :-/ All the matter is about PCM data. VGM format handle them in several possibles (and sometime really bad) ways... and XGM only support frame accurate timing which make PCM conversion a pain in some situation. I have to find a better method than the one i'm using (which become complex and not really accurate depending the situation).

[Stef]

Gasega68k>
I just uploaded a new version of my XGMTool to improve the VGM --> XGM conversion process.

https://github.com/Stephane-D/SGDK

Honestly it's still not perfect but definitely the VGM files you sent me are just insane :p The PCM data is built in really weird and not optimal way (many duplicated samples), some sample start at a point and continue to another... all that make the PCM samples extraction process really complex, even more as XGM is frame based timing when VGM format allow sample (1/44100 of second) based timing. Because of that, sometime some VGM will badly convert to XGM and result in faulty PCM instruments (incorrect pitch / incorrect sample length).
Also the sound driver used for Sonic 3 and Sonic 3D is also really weird, making again the PCM extraction painful... many genesis games suffers from main BUS contention issue when playing PCM data through Z80 and the XGMTool takes care of this and try to fix PCM samples. The problem with Sonic 3 is that the driver is designed to play PCM instrument this way :
- quickly send PCM data to DAC for a certain amount of time (~70% frame time)
- handle others stuff (no PCM for about ~30% frame time)
- and repeat that scheme again and again...

So XGMTool detects that PCM data is wrote at 19 Khz to the DAC and assume that is the wanted playback rate, but in fact no, the wanted playback rate is actually 19 Khz * 70% + 0 KHz * 30% so about ~14 Khz :-/
Of course i can easily assume this but that is a specific case which is in opposite statement to other games :-/

Also i updated the XGM driver itself, so you need to update your with the last z80_xgm.s80 file :
https://github.com/Stephane-D/SGDK/blob ... 80_xgm.s80

I fixed a stupid issue with the PCM play status which could affect some music play operation.

[gasega68k]

Thank you very much, Stef, I'll try it in a few moments.

I had thought that the problem could be in how sonic3d (I guess also sonic3) plays samples.
It's amazing that there is 30% of the frame time without PCM , it also explains the "noise" in the samples.

Again thank you very much.

[HardWareMan]

There are PCM only drivers. For example Minnesota Fats. Z80 plays only PCM and M68K take Z80 bus to send FM data to YM2612 directly every frame. So, they have artefacts in PCM sound too. Not 30% of frame time but very noticeable.

[Stef]

Thank you very much, Stef, I'll try it in a few moments.

I had thought that the problem could be in how sonic3d (I guess also sonic3) plays samples.
It's amazing that there is 30% of the frame time without PCM , it also explains the "noise" in the samples.

Again thank you very much.

You're welcome, that allows me to improve the xgmtool
Also when you meet a VGM file giving trouble to the converter, you can try to get another VGM dump of the same music (generated by a different tool) if possible, it should be enough to get a correct conversion process.Yeah definitely the sound driver in Sonic 3D & Sonic3 is weird when it comes to PCM stuff... it's indeed why PCM sounds so "graining" in these games.

There are PCM only drivers. For example Minnesota Fats. Z80 plays only PCM and M68K take Z80 bus to send FM data to YM2612 directly every frame. So, they have artefacts in PCM sound too. Not 30% of frame time but very noticeable.

Yeah a lot of games does that actually as it easier to do that way.
The problem is then more about DMA which can interrupt Z80 process for a longer time than the YM2612 68000 writes.
But i can detect these gaps during the conversion and fix the samples, the problem is that in Sonic 3 / 3D, the gap is definitely a lot longer than expected (longer than VBlank period) and so my tool was detecting several smalls samples instead of a big one separated by large gaps.

[NightShadow]

Hi


I don't post very often but as I'm in a project right now and I really need to ensure something before continuing...

First off, thank you very much stef for this music/sfx driver. I was hoping something like that as a solution to easily play sounds and musics.

I haven't tested it yet, as I haven't upgraded SGDK since...
I'll add the musics and sfx later in the development. maybe after 1 level is completed.

My game is a run N gun, very ambitious project, with many interesting features.. It's far from finish but I already coded the players gameplay, Now I need to advance with Maps and Collisions.. anyway, that's not the problem.

For some graphical effects, I constantly need to load some Tiles between frames... I do not use DMA ... My game will run at 30 fps and I can actually load 18 independant tiles to VDP between each frame and I can maintain my 30fps, above that it start to drops at 20 fps..
18 Tiles is what I need to be the minimum acceptable to work with my project.

I made a lot of running tests to overload the CPU to see if what I want to do is possible and I constantly need to find ways to optimize everything... In the end it should always run at 30 fps.

Now I would like to know.
When I'll be using the XGM driver, Will it affect anything about performance on what I'm doing with the 68000 ? Also the transfers, Can I assume I will be able to load my 18 Tiles rock solid or will it be slower as the Music payer already need to do loading transfers while playing ?


Thanks in advance for your help !

[Stef]

Hi NightShadow,

Are you loading tiles to VDP during VBlank period ? Because if you do it during active period transfer is much slower and even worst, the VDP will stalls the 68000 and Z80 if the later try to access the 68000 bus so that's definitely something to avoid.
Still 18 tiles per frame is definitely not much, with DMA you can upload about 150 tiles per frame and that's by keeping a bit of bandwidth for sprite table and tilemap update.

But to get back to your original question, using the XGM driver have a small impact on 68000 performance because of the 4 PCM channel which eat a bit of the 68000 bus time (14000 * 4bytes per second). The good thing is that impact is pretty stable as the 4 PCM mixing is always active even when no PCM are currently playing, also the performance impact is not that important : at maximum i measured a 4-5% performance drop on the 68000.

[NightShadow]

Hi again Stef

Thanks for fast reply,

5% on the 68000 is really acceptable for what the driver does ;-P I was hoping it wasn't around 20% or more... probably 68k players.

Now, I would just like to understand about DMA and you XGM driver, Can I use DMA + XGM ? I always though that playing music with DMA transfer wasn't working stable. Maybe it was for old music driver and not XGM ...

XGM player needs to load from Rom, right ? When I transfer Tiles from Rom to Vdp, is there any conflict or bandwith sharing with XGM playing tunes and sounds ? DMA or Not ?

If I can use DMA that'll be awesome, Now everything in my game was limited to the possible 18 Tiles I was working with, but if I can load more and even 150 per frame, it would be completely awesome, I could think about the possibility of doing a lot more things and effects than I though... even more ambitious project ;-P

[NightShadow]

oh, just forgot to mention ...

No I am not loading the 18 tiles during Vblank, In fact actually I don't know exactly how to handle this. I'm loading tiles in extra performances test function, between code...

Must I do it before graphic data update or after, or something like that ?

For now my code and tests works well, I have no bug of any kind right now... I would like to try loading during vblank, to see what I could do better, or even with DMA if it's possible.

[Stef]

Normally DMA have an impact on the PCM playback, the longer is the DMA and the worst is the playback quality. However the XGM driver should take care of this, with some limits though. Normally if you keep your DMA inside the VBlank area you shouldn't meet any troubles. In reality it depends from the XGM music loading, if the load is important then the PCM can suffers a bit but definitely less than a usual music driver. Still the XGM driver has an inconvenient : if the Z80 is stopped for a long time (more than 30% of it's usual running time) then the driver is not anymore able to handle the 4 PCM channel mixing + music playing and you will experience music slowdown.