Best looking palette fade - what's your opinion?

For anything related to VDP (plane, color, sprite, tiles)

Moderators: BigEvilCorporation, Mask of Destiny

Post Reply
powerofrecall
Very interested
Posts: 237
Joined: Fri Apr 17, 2009 7:35 pm
Location: USA

Best looking palette fade - what's your opinion?

Post by powerofrecall » Thu Jan 09, 2014 1:37 am

The MD's limited colorspace made different programmers tackle implementation of palette fade ins/outs in a few different ways--it seems like some of them look nice and smooth and some are really jerky and only fade a couple of levels before hitting black or just look bad.

Travellers' Tales games do a fade where the blue element is the last one faded out. It looks smooth, but is kind of obvious as well. So do the Sonic games, but it looks slightly different, like the green and red parts aren't faded out completely before fading the blue (same idea but staggered slightly). Most games seem to do what I imagine is basically just subtracting $200, $20, $2 from each palette entry in turn and repeating for each color until all the palette lines are zeroed out. This looks OK and consistent for bright and colorful graphics but it's also obvious that the darker colors on the MD are bigger "steps" than the bright ones. Games that can trace their lineage to early Sega Interactive/Interactive Designs like TaleSpin, Home Alone 2, Eternal Champs, Dinos for Hire etc have an absolutely disgusting looking fade that seems to somehow look washed out and gray. Ugh. There's probably a lot more examples of good and bad but those stuck out for me.

So what I'm getting at, and asking you, a board full of talented bastards like yourselves, is how would you tackle this problem to produce a clean fade to black/fade from black with the smoothest result? I feel like the real answer would also involve actually taking luminance and color theory into account and selectively fading with the whole palette accounted for but I'm curious what ideas you guys have come up with.

tomaitheous
Very interested
Posts: 256
Joined: Tue Sep 11, 2007 9:10 pm

Post by tomaitheous » Thu Jan 09, 2014 5:10 am

PCE has 9bit RGB very similar to the Genesis. I've done palette fade routines on it, but it's usually just subtract 1 from the R/G/B element and check for clipping. It's works, but yeah - it's not exactly the natural-est of looking fade in/out.

Other than precalc palettes (which some games use). You could always do RGB to YUV in realtime, decrement Y, then convert back to RGB. It's not fast, but if all you're doing it fading in or out for an area that has no player control (end of stage/beginning of stage), it should work fine. You can always optimize by converting once to YUV, and then have the routine decrement Y then convert to RGB (so you only do a convert once per Y increment or decrement).

TmEE co.(TM)
Very interested
Posts: 2440
Joined: Tue Dec 05, 2006 1:37 pm
Location: Estonia, Rapla City
Contact:

Post by TmEE co.(TM) » Thu Jan 09, 2014 8:43 am

I just decrement all colors equally, with some spacing. It looks nicer to me than if things are spread out over the colors. Drawback is that you can only have short fades, but I am fine with that.
Mida sa loed ? Nagunii aru ei saa ;)
http://www.tmeeco.eu
Files of all broken links and images of mine are found here : http://www.tmeeco.eu/FileDen

powerofrecall
Very interested
Posts: 237
Joined: Fri Apr 17, 2009 7:35 pm
Location: USA

Post by powerofrecall » Sat Jan 11, 2014 3:45 am

I'm hacking up a test/demo of calculating luma based on the palette entries and I've come up with a decent integer-only/add-and-shift only approximation of the CCIR601 formula to derive luma (Y) but I'm kind of stuck as to how to implement this as a way to help create a clean looking fade. It does however make a nice way to calculate a good even looking grayscale version of an arbitrary palette so I have that going. If I come up with anything cool I'll post it.

edit: I'm assuming I'll have to calculate U/V or Cb/Cr as well. I've got more studying to do then!

tomaitheous
Very interested
Posts: 256
Joined: Tue Sep 11, 2007 9:10 pm

Post by tomaitheous » Sat Jan 11, 2014 4:23 am

The PC-Engine's VCE chip (which houses the color ram and outputs composite and rgb video), bit packs the 9bit RGB and uses that to index a 512 entry LUT.

You can do this approach in software as well, but the problem is converting back. You still have to convert back to R/G/B. 9bit might be enough for a RGB->YUV table conversion, but YUV will need more than 3bits per element to accurately represent those RGB values in YUV space. Thus you can't just use YUV as a single index to convert back from a different table. PCE does 15bit YUV and there are errors in this (steps). 15bits would be a 32k WORD table (64kbytes). That a lot to waste on a simple fade routine, not to mention the accuracy issues you'll run into with 15bit YUV (though for fades, it might not that noticeable). I don't recommend doing any of this...

And by YUV, I mean Y, R-Y, B-Y.

Instead, check out this page: http://www.fourcc.org/fccyvrgb.php

Look at the constants (coefficients) in the above formula. Since they are constant/fixed in value, you make a handful of small LUTs to handle these (multiplication LUT for each coefficient value), and make the RGB conversion fairly fast. For RGB to YUV, these LUTs are small (8 entries each since the base R/G/B element is 3bits). Same with the conversion back, but Y/U/V should be 8bit values, so the LUTs are gonna be a little bit bigger - but nothing to break the bank over.

Of course, you can convert to other formats as well (HSL or HSV).

If you're not sure about how any of this works or such, try it out on PC first. I.e. Write a quick PC app for these conversions and mess with the values in the converted space, then convert back to RGB.

powerofrecall
Very interested
Posts: 237
Joined: Fri Apr 17, 2009 7:35 pm
Location: USA

Post by powerofrecall » Sat Jan 11, 2014 5:03 am

I see where you're going with this. I'll admit my understanding of color spaces isn't the greatest (doesn't really extend beyond RGB). You're right about converting the YUV back to RGB--I figured as much going in. What I had assumed in particular though is having a separate luma would ease calculating the actual relative brightness of palette entries. Here's what I'm thinking though:

If we're talking 9 bit RGB with a whole 3 bits per color, we won't need too much precision--certainly not multiple decimal places worth. I am currently calculating Y for instance off the 9 bit values this way

Y = (R+R+R+B+G+G+G+G) >> 3

which is certainly crude (found it on stackoverflow I believe in a google search) but does have the advantage of being calculated quickly and color weighted to an actual model. I figure it is acceptable considering the limited palette. It produces a result that seems to fit into 4 bits which I am masking off to make greyscale palettes as a nice side effect. Hopefully I can work out a simple integer based approximation for U and V as well but I feel I am still going to have my work cut out for me in processing the YUV result and back into RGB. Of course it's entirely possible this is just way too complicated to pursue even for a better looking fade, but I still want to give it a real try. :)

Thanks for the info, I'll be poring over it and seeing what I can do.

powerofrecall
Very interested
Posts: 237
Joined: Fri Apr 17, 2009 7:35 pm
Location: USA

Post by powerofrecall » Sat Jan 11, 2014 6:25 pm

I'm starting to think that I'm overthinking it. Now that I've successfully figured a way to get an overall brightness value (somewhat) perceptually weighted to color, I can probably just use that to make a "weighted" conventional fade instead of doing the whole dance of calculating out the whole color space. Something like calculating Y for the whole palette, then decrementing the brightest colors on each pass?

Either that or I'll just scrap the whole thing and keep it simple. :D

Post Reply