Problems with Highlight Shadow Mode

SGDK only sub forum

Moderator: Stef

Post Reply
themrcul
Very interested
Posts: 116
Joined: Fri Apr 15, 2016 2:21 pm

Problems with Highlight Shadow Mode

Post by themrcul » Wed Jan 11, 2017 4:10 am

Hi all,
Now that I have scrolling working in my little engine I am experimenting with Mega Drive graphics before I make plans for a game.
I am planning on a top/down game, so using highlight/shadow mode is a possibility!

However, from copious reading of the docs (one at https://emudocs.org/Genesis/Graphics/genvdp.txt, another at http://md.squee.co/VDP#Shadow.2FHighlight), and forum posts, the behaviour I have found in emulators and on real console is different from what I expected.

In my example, I have 2 layers of tilemaps, both seem to be working well for priority - with priority, they are normal intensity, without, they are shadowed.

Off to the side of the screen I have two sprites, one is a 2x2 tile high priority sprite which just contains fills of the first few colours palette 0.
On top of that sprite I have another sprite, which I will call the "modifier sprite", which uses the same tileset, but in palette 4, to attempt to get highlight and shadow effects onto the sprites and tiles below that sprite (using colours 15 and 16).

However it is with this modifier sprite that I have found problems.

On the Gens emulator, using any colour index in palette 4 from 1 to 14 is fine. Please see below screenshot of using palette index 14 (a blue tone), in the middle of the 2x2 tile sprite below it.
ColourIndex14.png
ColourIndex14.png (10.71 KiB) Viewed 6185 times
Using colour index 15 to highlight pixels underneath successfully highlights the sprite pixels underneath the modifier sprite.
Highlight.png
Highlight.png (10.31 KiB) Viewed 6185 times
Using colour index 16 to shadow the pixels underneath doesn't display the modifier sprite at all, as can be seen below:
Shadow.png
Shadow.png (9.64 KiB) Viewed 6185 times
Please see post below for Kega screenshots.

themrcul
Very interested
Posts: 116
Joined: Fri Apr 15, 2016 2:21 pm

Re: Problems with Highlight Shadow Mode

Post by themrcul » Wed Jan 11, 2017 4:14 am

Using the Kega Fusion Emulator has the same result as real hardware, so I will show screenshots of Kega Fusion below:

Using colour index 14 is the same as Gens, it produces a blue colour as per the palette loaded in palette slot 4.

Using colour index 15 to highlight or index 16 to shadow appears to mask the sprite below it (like a cutout), and it will then highlight/shadow the tilemap layers below it:
Shadow:
Shadow2.png
Shadow2.png (7.34 KiB) Viewed 6184 times
Highlight:
Highlight2.png
Highlight2.png (7.27 KiB) Viewed 6184 times

Nowhere have I read that highlight/shadow colours on sprites would cutout the sprites below it and only show the tilemaps below. Is there something I am doing wrong?

Please see my code below, in this case it is a single sprite that should shadow the sprites and tilemaps below it. It is on hardware cutting out the 4-fill sprite below it and shadowing the tilemaps below the sprite instead:

Code: Select all

int main()
{
	// Initialize the video processor, set screen resolution to 320x224
	VDP_init();
	VDP_setScreenWidth320();
	if (IS_PALSYSTEM)
		VDP_setScreenHeight240();
	else
		VDP_setScreenHeight224();

	// Initialize the controllers and set up the controller callback function
	JOY_init();
	JOY_setEventHandler(NULL);

	// Set up VDP modes
	VDP_setHilightShadow(1);
	VDP_setPlanSize(64, 32);
	VDP_setScrollingMode(HSCROLL_PLANE, VSCROLL_PLANE);
	
	// put sprite high priority in scene
	VDP_setSprite(1, 140, 140, SPRITE_SIZE(2, 2), TILE_ATTR_FULL(0, 1, 0, 0, 2));
	VDP_setSprite(0, 144, 144, SPRITE_SIZE(1, 1), TILE_ATTR_FULL(3, 1, 0, 0, 15));
	VDP_linkSprites(0, 1);
	VDP_updateSprites(2, 0);

	while (1)
	{
		VDP_waitVSync();
	}
}

Mask of Destiny
Very interested
Posts: 615
Joined: Thu Nov 30, 2006 6:30 am

Re: Problems with Highlight Shadow Mode

Post by Mask of Destiny » Wed Jan 11, 2017 6:45 am

The reason for the behavior you're seeing is that shadow/highlight is handled during the final output phase, not during sprite rendering. Sprite rendering is the very first step in drawing a line and primarily happens during HBlank and the border area. Sprites are rendered to an internal line buffer that stores 7 bits of information per pixel: a 4 bit color index, a 2 bit palette index and a priority bit. Since sprite rendering doesn't know anything about shadow and highlight, the special "operator" colors are treated like opaque pixels overwriting any pixels of sprites below them.

Later in the line, this line buffer is combined with the background layers. It's at this point that the operator colors have special handling. They are no longer opaque and instead alter the shadow/highlight status of whatever is below it. Since the sprite have already been flattened at this point, the only thing that can be below the operator pixels are background tiles.

themrcul
Very interested
Posts: 116
Joined: Fri Apr 15, 2016 2:21 pm

Re: Problems with Highlight Shadow Mode

Post by themrcul » Wed Jan 11, 2017 11:20 am

Thank you Mask for that informative reply.
That explains why I haven't seen more games use highlight/shadow mode!

So with sprites, the best that can be done is per-tile shadowing using the low priority flag. Disappointing, but understandable! :(

Thanks again Mask. :)

Stef
Very interested
Posts: 3131
Joined: Thu Nov 30, 2006 9:46 pm
Location: France - Sevres
Contact:

Re: Problems with Highlight Shadow Mode

Post by Stef » Wed Jan 11, 2017 3:53 pm

Oh i didn't know that, i always though highlight / shadow operator could work on sprite as well (that explain Gens behavior). So that is not the case... that is a bit disappointing indeed :-/

Mask of Destiny
Very interested
Posts: 615
Joined: Thu Nov 30, 2006 6:30 am

Re: Problems with Highlight Shadow Mode

Post by Mask of Destiny » Thu Jan 12, 2017 1:18 am

Stef wrote:Oh i didn't know that, i always though highlight / shadow operator could work on sprite as well (that explain Gens behavior). So that is not the case... that is a bit disappointing indeed :-/
Shadow/Highlight is indeed a very limited feature, though I imagine it took relatively few transistors to implement as a result. The documentation for it has also been less than stellar so the confusion is understandable.

astrofra
Interested
Posts: 28
Joined: Sun Dec 14, 2014 8:50 am
Location: Orleans | France
Contact:

Re: Problems with Highlight Shadow Mode

Post by astrofra » Sat Jan 14, 2017 7:19 am

Very interesting discussion, thanks for the technical explanation!
The way sprites are linearly processed absolutely makes sense, considering the age of the hardware.

I wouldn't say it is *that* limited, though.

Admittedly, sprites cannot mutually overshadow.
But a series of game sprites can easily cast shadows on the ground, achieving a very consistent result. That's what the MD was built for in the first place, wasn't it ? :)

Example :
Image
640 polygons are enough for everyone.

Stef
Very interested
Posts: 3131
Joined: Thu Nov 30, 2006 9:46 pm
Location: France - Sevres
Contact:

Re: Problems with Highlight Shadow Mode

Post by Stef » Sat Jan 14, 2017 10:46 am

Your demo is indeed a good use of the shadow effect ;)
And you're right, it was supposed to be used like this i guess, kind of shadow/hilight operator on background ;)
The thing that make it complicated to use is that it also works with priority... because of that many games cannot use it. Having only sprites acting as H/S operators (with 62 & 63 entries) would make this feature far easier to use (but a bit more limited in some cases though).

Post Reply