CVSD Compressed audio example

Announce (tech) demos or games releases

Moderator: Mask of Destiny

Chilly Willy
Very interested
Posts: 2762
Joined: Fri Aug 17, 2007 9:33 pm

Post by Chilly Willy » Fri Jul 23, 2010 2:44 am

Yeah, it'll need a little converting. If you don't have it yourself by later today, I'll do the conversion myself. I just compiled the 1.07 rc7 code and am interested in how it does.

EDIT: Okay, here's musicdemo4 with the Z80 code and the makefile altered for sjasmplus. It was really simple.

http://www.fileden.com/files/2009/2/3/2 ... icdemo7.7z

EDIT 2: In case people were wondering, the pcm2tadpcm file takes a raw sound as the input, and outputs a compressed binary file. The input should have a sample rate of 22050 Hz, and be mono 16 bit little-endian PCM samples. The compression ratio compared to the input is 6:1, but since you'd never store 16 bit pcm on the MD, the effective compression ratio is 3:1.

BinaryCreature
Newbie
Posts: 7
Joined: Thu Jul 22, 2010 3:38 am

Post by BinaryCreature » Fri Jul 23, 2010 1:17 pm

Chilly Willy wrote:Yeah, it'll need a little converting. If you don't have it yourself by later today, I'll do the conversion myself. I just compiled the 1.07 rc7 code and am interested in how it does.

EDIT: Okay, here's musicdemo4 with the Z80 code and the makefile altered for sjasmplus. It was really simple.

http://www.fileden.com/files/2009/2/3/2 ... icdemo7.7z

EDIT 2: In case people were wondering, the pcm2tadpcm file takes a raw sound as the input, and outputs a compressed binary file. The input should have a sample rate of 22050 Hz, and be mono 16 bit little-endian PCM samples. The compression ratio compared to the input is 6:1, but since you'd never store 16 bit pcm on the MD, the effective compression ratio is 3:1.
Thank you. I am hitting some speedbumps setting up the toolchain on windows. Can you recommend toolchain components I must install to make this compile ? A template map/description of somebody's successfully installed toolchain would help. Thanks.

Chilly Willy
Very interested
Posts: 2762
Joined: Fri Aug 17, 2007 9:33 pm

Post by Chilly Willy » Fri Jul 23, 2010 8:23 pm

You should be able to use Stef's Mini DevKit. The libmd in the demo is just his mini devkit made into a library instead of as separate files.

Here's a new compile of the library... you can copy it over by hand into your own toolchain.

http://www.fileden.com/files/2009/2/3/2 ... 0100723.7z

I put the includes in $HOME/Tools/gendev/m68k/sdk/include, and the lib in $HOME/Tools/gendev/m68k/sdk/lib. You can put it someplace else, and change the paths in the demo makefile.

The compiler in Stef's mini devkit are getting kinda old - I suggest you replace the 68000 compiler with the one from MINT (the Atari ST replacement OS). KPIT has the latest toolchain for the SH processor (if you wish to do 32X programming).

You can find the MINT toolchain here: http://vincent.riviere.free.fr/soft/m68k-atari-mint/

The cygwin packages have the Windows binaries.

sega16
Very interested
Posts: 251
Joined: Sat Jan 29, 2011 3:16 pm
Location: U.S.A.

how would this work in asm

Post by sega16 » Sat Jan 29, 2011 3:21 pm

I tried and tried to get this to work in asm but no matter what I do it does not work.Here is my code anyone know what I am doing wrong:
;---------------------------------------
;
; Sega Mega Drive code template v1.0
;
; by drx [www.hacking-cult.org]
; [www.bluehedgehog.org]
; [www.shadowsoft-games.com]
;
; Inits Z80, PSG, VDP, RAM and does the security stuff
; You put your code under label Main
;
;---------------------------------------


Vectors: dc.l $FFFE00, Entrypoint, Error, Error
dc.l Error, Error, Error, Error
dc.l Error, Error, Error, Error
dc.l Error, Error, Error, Error
dc.l Error, Error, Error, Error
dc.l Error, Error, Error, Error
dc.l Error, Error, Error, Error
dc.l HBlank, Error, VBlank, Error
dc.l Error, Error, Error, Error
dc.l Error, Error, Error, Error
dc.l Error, Error, Error, Error
dc.l Error, Error, Error, Error
dc.l Error, Error, Error, Error
dc.l Error, Error, Error, Error
dc.l Error, Error, Error, Error
dc.l Error, Error, Error, Error
Header: dc.b 'SEGA MEGA DRIVE ' ; Console name
dc.b '(C) DRX 2004.NOV' ; Copyright/Date
DomesticName: dc.b 'Code template by drx - www.hacking-cult.org ' ; Domestic Name
dc.b 'Code template by drx - www.hacking-cult.org ' ; International Name
dc.b 'GM 00000000-00' ; Version
Checksum: dc.w $1337
; Checksum
dc.b 'J ' ; I/O Support
RomStartLoc: dc.l 0 ; ROM Start
RomEndLoc: dc.l RomEnd
; ROM End
RamStartLoc: dc.l $FF0000 ; RAM Start
RamEndLoc: dc.l $FFFFFF ; RAM End

dc.b $20,$20,$20,$20 ; 'RA',$F8,$20 if SRAM = on

SramStart: dc.l $20202020 ; $200000 if SRAM = on
SramEnd: dc.l $20202020 ; $20xxxx if SRAM = on

dc.b ' ' ; Notes
dc.b ' '
dc.b 'JUE ' ; Country

;---------------------
; Code start
;---------------------

Entrypoint:

tst.l ($A10008).l ;Test Port A control
bne PortA_Ok

tst.w ($A1000C).l ;Test Port C control

PortA_Ok:
bne SkipSetup

move.b ($A10001).l,d0 ;version
andi.b #$F,d0
beq SkipSecurity ;if the smd/gen model is 1, skip the security
move.l #'SEGA',($A14000).l

SkipSecurity:

move.w ($C00004).l,d0 ;test if VDP works

moveq #0,d0
movea.l d0,a6
move.l a6,usp ;set usp to $0

;---------------------
; Setup VDP registers
;---------------------
lea (VDPSetupArray),a0
move.w #(VDPSetupArrayEnd-VDPSetupArray)/2,d1 ;$18 VDP registers

VDPSetupLoop:
move.w (a0)+,($C00004).l
dbf d1,VDPSetupLoop


move.l #$40000080,($C00004).l
move.w #0,($C00000).l ;clean the screen


;---------------------
; Init the Z80
;---------------------

move.w #$100,($A11100).l ;Stop the Z80
move.w #$100,($A11200).l ;Reset the Z80

Waitforz80:
btst #0,($A11100).l
bne Waitforz80 ;Wait for z80 to halt

lea (Z80Init),a0
lea ($A00000).l,a1
move.w #Z80InitEnd-Z80Init,d1

InitZ80:
move.b (a0)+,(a1)+
dbf d1,InitZ80

move.w #0,($A11200).l
move.w #0,($A11100).l ;Start the Z80
move.w #$100,($A11200).l


;---------------------
; Reset the RAM
;---------------------

lea ($FFFF0000).l,a0
move.w #$3fff,d1

ClearRAM:
move.l #0,(a0)+
dbf d1,ClearRAM


;---------------------
; VDP again
;---------------------

move.w #$8174,($C00004).l
move.w #$8F02,($C00004).l


;---------------------
; Clear the CRAM
;---------------------

move.l #$C0000000,($C00004).l ;Set VDP ctrl to CRAM write
move.w #$3f,d1

ClearCRAM:
move.w #0,($C00000).l
dbf d1,ClearCRAM


;---------------------
; Clear the VDP stuff
;---------------------

move.l #$40000010,($C00004).l
move.w #$13,d1

ClearStuff:
move.l #0,($C00000).l
dbf d1,ClearStuff


;---------------------
; Init the PSG
;---------------------

move.b #$9F,($C00011).l
move.b #$BF,($C00011).l
move.b #$DF,($C00011).l
move.b #$FF,($C00011).l


move.w #0,($A11200).l


;---------------------
; Load the z80 driver
;---------------------

move.w #$100,($A11100).l ;Stop the Z80
move.w #$100,($A11200).l ;Reset the Z80

Waitforz80a:
btst #0,($A11100).l
bne Waitforz80a ;Wait for z80 to halt

lea (Z80Driver),a0
lea ($A00000).l,a1
move.W #Z80DriverEnd-Z80Driver,d1

LoadZ80Driver:
move.b (a0)+,(a1)+
dbf d1,LoadZ80Driver

move.w #0,($A11100).l ;Start the Z80

;---------------------
; Clear the registers
; and set the SR
;---------------------

movem.l ($FF0000).l,d0-a6
lea ($FFFE00).l,a7
move #$2700,sr

SkipSetup:


;-----------------------
; Here starts your code
;-----------------------
Main:

;jsr Sound_E4
; jsr PCMDriverLoad
;the code already loads the driver
move.w #$100,($A11100).l ;Stop the Z80
WaitZ80_81:
btst #0,($A11100).l
bne.s WaitZ80_81 ;Wait for z80 to stop
;move.b #1,($a00039).l ;turn the driver on
move.b #(((Music81)&$FF0000)>>16),($a00042).l ;location - $__xxxx
move.b #(((Music81)&$FF00)>>8),($a00041).l ;location - $xx__xx
move.b #((Music81)&$FF),($a00040).l ;location - $xxxx__
move.b #(((Music82-Music81)&$FF0000)>>16),($a00045).l ;length - $__xxxx
move.b #(((Music82-Music81)&$FF00)>>8),($a00044).l ;length - $xx__xx
move.b #((Music82-Music81)&$FF),($a00043).l ;length - $xxxx__
move.b #$C0,($a0004D).l ;pan center
move.b #$01,($a0004E).l ;play how many times
move.b #$00,($a0004F).l ;pause = 0
move.w #$0,($A11100).l ;Start the Z80
;rts
loop:
bra loop
rts


;---------------------
; Error exceptions
;---------------------

Error:
rte

;---------------------
; Horizontal Blank
;---------------------
HBlank:

rte

;---------------------
; Vertical Blank
;---------------------
VBlank:

rte

;---------------------
; VDP registers array
;---------------------

VDPSetupArray:
dc.w $8004 ;9-bit palette = 1 (otherwise would be 3-bit), HBlank = 0
dc.w $8134 ;Genesis display = 1, DMA = 1, VBlank = 1, display = 0
dc.w $8230 ;Scroll A - $C000
dc.w $8338 ;Window - $E000
dc.w $8407 ;Scroll B - $E000
dc.w $857c ;Sprites - $F800
dc.w $8600 ;Unused
dc.w $8700 ;Backdrop color - $00
dc.w $8800 ;Unused
dc.w $8900 ;Unused
dc.w $8A00 ;H Interrupt register
dc.w $8B00 ;Full screen scroll, no external interrupts
dc.w $8C81 ;40 cells display
dc.w $8D3F ;H Scroll - $FC00
dc.w $8E00 ;Unused
dc.w $8F02 ;VDP auto increment
dc.w $9001 ;64 cells scroll
dc.w $9100 ;Window H position
dc.w $9200 ;Window V position
dc.w $93FF ;DMA stuff (off)
dc.w $94FF ;DMA stuff (off)
dc.w $9500 ;DMA stuff (off)
dc.w $9600 ;DMA stuff (off)
dc.w $9780 ;DMA stuff (off)
VDPSetupArrayEnd:


;---------------------
; Z80 init code
;---------------------

Z80Init:
dc.w $af01, $d91f, $1127, $0021, $2600, $f977
dc.w $edb0, $dde1, $fde1, $ed47, $ed4f, $d1e1
dc.w $f108, $d9c1, $d1e1, $f1f9, $f3ed, $5636
dc.w $e9e9
Z80InitEnd:

;---------------------
; Music driver (z80)
;---------------------

Z80Driver:
INCBIN z80_tadpcm.bin
even
Z80DriverEnd:
Music81: incbin song1.snd
even
music82:

RomEnd:
I just can not seem to get this to work.

Chilly Willy
Very interested
Posts: 2762
Joined: Fri Aug 17, 2007 9:33 pm

Post by Chilly Willy » Sat Jan 29, 2011 10:19 pm

Always use code tags with code!

Not sure what assembler you're using, but your header is wrong - you don't have enough spaces in several places. Here's a proper header:

Code: Select all

Header:
    dc.b "SEGA MEGA DRIVE " ; Console name
    dc.b "(C) DRX 2004.NOV" ; Copyright/Date
DomesticName:
    dc.b "Code template by drx - www.hacking-cult.org     " ; Domestic Name
    dc.b "Code template by drx - www.hacking-cult.org     " ; International Name
    dc.b "GM 00000000-00" ; Version
Checksum:
    dc.w $1337  ; Checksum
    dc.b "J               " ; I/O Support
RomStartLoc:
    dc.l 0 ; ROM Start
RomEndLoc:
    dc.l RomEnd ; ROM End
RamStartLoc:
    dc.l $FF0000 ; RAM Start
RamEndLoc:
    dc.l $FFFFFF ; RAM End

    dc.b $20,$20,$20,$20 ; 'RA',$F8,$20 if SRAM = on

SramStart:
    dc.l $20202020 ; $200000 if SRAM = on
SramEnd:
    dc.l $20202020 ; $20xxxx if SRAM = on

    dc.b  "    "
    dc.b  "                "
    dc.b  "                "
    dc.b  "                "
    dc.b  "F               "  ; enable any hardware configuration
Always check the binary of the finished file in a hex editor to make sure everything is at the proper place in the header. The exception table should go from 0 to $FF, and the header from $100 to $1FF, and the code should start right at $200. The "F" is the new world code to use in place of JUE.

You also aren't giving the Z80 time enough to init before you try to start playing the data. You need to waste some time so the z80 has a chance to clear its vars, setup the ym2612, and then enter the main loop. At main, put

Code: Select all

;---------------------------
; Wait for Z80 driver to run
;---------------------------
    move.w #20000,d0
waste_time:
    nop
    dbra d0, waste_time
TADPCM data also needs to be four byte aligned, not two.

Chilly Willy
Very interested
Posts: 2762
Joined: Fri Aug 17, 2007 9:33 pm

Post by Chilly Willy » Sat Jan 29, 2011 11:57 pm

Here's sega16's demo with fixes and changes to some comments. I assembled it with the ASL Macroassembler:

http://john.ccac.rwth-aachen.de:8000/as/

To assemble:

Code: Select all

asl -cpu 68000 -L -o ademo.p ademo.s
p2bin -r \$00000-\$FFFFF ademo
That assumes you added the path to the asl binaries to your PATH environment variable like this:

Code: Select all

export PATH=/home/jlfenton/Tools/gendev/asl/bin:$PATH
ademo.zip

That includes the fixed ademo.s file, the z80 object code from my musicdemo7, a Sonic CD track in the proper compressed format from the same demo, and the final binary.

sega16
Very interested
Posts: 251
Joined: Sat Jan 29, 2011 3:16 pm
Location: U.S.A.

Post by sega16 » Mon Jan 31, 2011 1:46 am

Thank you all for your help.It worked perfect!

Chilly Willy
Very interested
Posts: 2762
Joined: Fri Aug 17, 2007 9:33 pm

Post by Chilly Willy » Mon Jan 31, 2011 1:59 am

sega16 wrote:Thank you all for your help.It worked perfect!
You're welcome. It's just good to see someone taking an interest in this. If you need any other help, just ask. 8)

amushrow
Interested
Posts: 44
Joined: Mon Jan 02, 2017 12:56 pm

Re: CVSD Compressed audio example

Post by amushrow » Fri Dec 28, 2018 2:26 am

Revivification!

I've been using my free time over Christmas to work on some software stuff before I'm extra busy again in January and have been looking into audio compression for on the MD. After a while I came across this stuff, unfortunately all the fileden links are dead so I've not been able to compare against any previous work, but I did find TmEE's notes on their idea and used that as a starting point (TADPCM.JPG). I also found the pcm2tadpcm etc. binaries on The Devster Forum which I used to check what the output is like, but it seems to have an issue with quite sounds which get removed entirely (and I'm also not sure if those files match the original implementation here).

Was there ever any more work on this? I see a few references to tadpcm around the place but only with dead links to stuff (I know it was quite a while ago).
I've gone as far as I can and have a working algorithm which should be super easy for the Z80 to run, but before I go through the rigmarole of getting it running on the MD I wanted to compare what I have with stuff that's already floating about. If I get it running on the console I'll start a new thread, but I'll link some audio samples I have so far. Compression (compared to an 8-bit PCM) are 33% for the High setting, 27% for the Low setting and 25% for the Ultra Low setting.

Samples
There are no tools in there to encode/decode new samples as I'm very much still pissing about with it, and the encoder is currently super dumb and slow.

Chilly Willy
Very interested
Posts: 2762
Joined: Fri Aug 17, 2007 9:33 pm

Re: CVSD Compressed audio example

Post by Chilly Willy » Fri Dec 28, 2018 4:14 pm

Still finding stuff that I haven't reposted since fileden went away...

Here's most of my MD audio demos.

http://www.mediafire.com/file/6rleye15p ... os.7z/file

Note that the VQ demos don't have player code for the MD yet... it's still the AMBTC code. The encoder and decoder work fine, but I haven't gotten around to the Z80 driver yet. I really should - VQ was about the best compression for old 8-bit systems.

Miquel
Very interested
Posts: 446
Joined: Sat Jul 30, 2016 12:33 am

Re: CVSD Compressed audio example

Post by Miquel » Fri Dec 28, 2018 5:50 pm

Chilly Willy has proven already to be a no shit guy.
Many thanks for your solid contributions man!

Chilly Willy
Very interested
Posts: 2762
Joined: Fri Aug 17, 2007 9:33 pm

Re: CVSD Compressed audio example

Post by Chilly Willy » Fri Dec 28, 2018 11:24 pm

You're welcome. In case anyone is interested in using the VQ formats, they're a codebook of 256 entries of four or eight 8-bit samples, followed by a buffer of codes for the sample.

VQ2: 2KB per buffer - 256 x 4 samples followed by 1KB of codes. 2:1 compression.
VQ3: 4KB per buffer - 256 x 4 samples followed by 3KB of codes. 3:1 compression.
These sound awesome - slightly more noise than ADPCM with the same compression, but much better frequency response.

VQ4: 4KB per buffer - 256 x 8 samples followed by 2KB of codes. 4:1 compression.
VQ6: 8KB per buffer - 256 x 8 samples followed by 6KB of codes. 6:1 compression.
These are noticeably noisier, but not bad, and have awesome compression rates.

I designed all VQ schemes to use a power of two size buffers to make easier to fit in the 32KB rom space on the Z80 side. Each code is simply an index into the codebook, and you play each sample of the entry in sequence at the desired sample rate. Super-simple playback, the main pro for VQ compression. The effort is all in the encoder. The encoder is derived from the KOS VQ texture compressor that's part of the Dreamcast KallistiOS tools. The compressors all accept a high quality switch to do extra passes over the sound quantizing, but in my testing, I don't notice much of a difference. On a modern PC on sound samples, it's pretty fast encoding, but I could see where this would have been agonizingly slow on PCs back when this type of compression was more relevant. The comment in the compressor says
You need 22050 Hz mono unsigned 8-bit pcm as the input.
To convert: ffmpeg -i in.ext -c:a pcm_u8 -ar 22050 -ac 1 -f u8 out.raw
but it really can use any sample rate at all. That's from when I was getting ready to convert the ambtc player code, which was hardcoded for 22050 Hz playback. Given how simple this compression is, it should be easier to make higher speed and/or multiple rate output routines. Note that the encoding is done on fixed length buffers, so the input sample will be padded (with silence) to the buffer size when you reach the end of the sample. For really short samples, you might want to concatenate the sounds to avoid wasted space from the buffer padding. Just be sure that separate samples are padded to the codebook entry size (four or eight samples) to avoid having samples mixed in the same entry.

Or simply use my compressor/decompressors as an example and make your own format. That's the nice thing about open source - changing things to suit your own purposes. 8)

EDIT: Want to hear the difference in the vq formats? Here's two songs encoded into each format, then converted back to pcm.
http://www.mediafire.com/file/ikwrj7f1k ... ps.7z/file

You can play them using ffplay with the line

Code: Select all

ffplay -f u8 -ar 22050 -ac 1 file.raw

cero
Very interested
Posts: 255
Joined: Mon Nov 30, 2015 1:55 pm

Re: CVSD Compressed audio example

Post by cero » Sat Dec 29, 2018 10:17 am

I listened to your YT video from 2013, and even the best compressed sample was terrible compared to the baseline... Have things improved since that video?

Chilly Willy
Very interested
Posts: 2762
Joined: Fri Aug 17, 2007 9:33 pm

Re: CVSD Compressed audio example

Post by Chilly Willy » Sat Dec 29, 2018 2:49 pm

CVSD was bad... AMBTC is worse, but VQ is actually really good. Check the pcm files I posted last night. I included two tracks of music converted to each of my VQ formats and then back to pcm so you can see how the compression affected them. 2:1 is really good, 3:1 is good, 4:1 isn't bad, and even 6:1 isn't that bad. They're all much better than CVSD or AMBTC.

Granted, I could make my CVSD examples sound better by filtering them based on the sample rate desired. I don't bother trying to make things sound better for my examples.

cero
Very interested
Posts: 255
Joined: Mon Nov 30, 2015 1:55 pm

Re: CVSD Compressed audio example

Post by cero » Sat Dec 29, 2018 5:20 pm

The youtube video was "AMBTC VQ Audio Demonstration", is that a different VQ? I tried your samples from the prev post, and I think they still sound bad. Bad Apple starts ok but becomes terrible when the vocals start, Harry Potter starts bad, both at the best vq2 version. The higher-compressed ones are clearly worse though.

edit: A VQ approach in general should be good, so this is perhaps the encoder being limited. I haven't looked at the source, but I guess it goes by the usual vector distance of a single block. The relationship between blocks matters more in audio than in images, and it might be possible to use lame's psychoacoustic model for a better cost function.

Post Reply