MD VDP color levels
Posted: Tue Oct 20, 2015 5:45 am
I took the time and measured out the actual output of the MD/Genny VDP.
Emulators used to completely misemulate the output, using RGB step of 16 to get their values and special case the white so it is white not grey or leave it grey, and also shadow and highlight behaviour was incorrect also. It was assumed that highlight makes white whiter so that was assumed to be only way to get the actual white. That also meant the ramp has 16 values total.
Couple years ago I helped to verify the behaviour of the shadow and highlight mode and it came out there's only 15 levels. Some Emulators then updated their schit to show the highlight part correctly. White is white, it doesn't get whiter, so the step would be 18 or so. Nice and round and definitely more correct. The non linear ramp was also seen at that time but I don't think anyone really took the time to measure it out.
Now today I looked at the output in detail as I can and you can see the output is a bit tangential. Compressed in the middle, expanded on the sides. I also verified that Shadow or Highlight doesn't introduce any new levels, not even subtly different ones.
So here are my results :
Old incorrect linear levels :
Normal : 0 36 72 108 144 180 216 252/255
Shadow : 0 18 36 54 72 90 108 126
HiLite : 126 144 162 180 198 216 234 252/255
Nonlinear levels derived from hardware measurements :
Normal : 0 52 87 116 144 172 206 255
Shadow : 0 29 52 70 87 101 116 130
HiLite : 130 144 158 172 187 206 228 255
Red means value got higher, blue means it got lower compared to the linear ramp.
All MD models I tested (MD1, variety of MD2 ASICs, the one that is shared by MD2 and Genny3) seemed to have identical output, and all color channels have same levels too.
I did multiple measurements and these are the values I ended up on each time.
In the end you lose some precision in the extremes, which hurts the dark stuff most badly as far as art goes.
Next up would be to measure the output in SMS mode, and also from real SMS and see if there's a difference.