Going Bonkers (VRAM reads)

For anything related to VDP (plane, color, sprite, tiles)

Moderators: BigEvilCorporation, Mask of Destiny

RetroRalph
Interested
Posts: 16
Joined: Wed Feb 24, 2010 6:13 am
Contact:

Going Bonkers (VRAM reads)

Post by RetroRalph » Thu Aug 26, 2010 2:37 am

One of my users reported that the MD game Bonkers had graphics corruption in RetroCopy so I took a look at it. RetroCopy's main aim is to emulate like hardware so I often run into weird bugs and undocumented areas. In this case VRAM reads. Bonkers seems to write to VRAM depending upon certain reads it gets from VRAM. Possibly some sort of caching mechanism it uses.

Firstly here is how RetroCopy looked with the bug :-

Image

You can notice the graphic corruption. This bug slipped my mind even though it's quite logical. Basically VRAM READS seem to be constricted by the same sort of VRAM accesses as WRITING. RetroCopy was not stopping reads when there was data in the FIFO so Bonkers is getting invalid data and tries to update it again, which runs into active display VRAM write limitations. The end result, corruption.

And here is how it looks if you either stop the 68K on reads when there is data in the FIFO (what RetroCopy does), or process the FIFO immediately by writing the value (what most emulators do).

Image

Anyhow I thought this might interest some of the emulator authors out there, as it should affect timing. Another interesting thing about this game is its SEGA intro. It has at least 3 different intros it uses, the weird thing is why it decides to use them. I noticed if I don't emulate FIFO correctly it gives me an exploding SEGA type logo when on the USA Genesis. Other regions (Japan/PAL) give different effects too. Maybe someone who is interested can RE it. I tried a few different emulators, Fusion, Gens-GX, Regen, and they seem to give different intros among each other too.

Image

Eke
Very interested
Posts: 884
Joined: Wed Feb 28, 2007 2:57 pm
Contact:

Post by Eke » Thu Aug 26, 2010 8:06 am

RetroCopy was not stopping reads when there was data in the FIFO so Bonkers is getting invalid data and tries to update it again, which runs into active display VRAM write limitations. The end result, corruption. And here is how it looks if you either stop the 68K on reads when there is data in the FIFO (what RetroCopy does), or process the FIFO immediately by writing the value (what most emulators do).
Interesting... I always thought that VRAM read were buffered (like with SMS VDP) and they would not hang the CPU. I remember Nemesis also wondered how the read were behaving regarding the FIFO but I think it's quite logical that way: first write cycles are performed from the FIFO then read cycle if any, which mean that if you do a write immediately followed by a read, it would hang the CPU until the write is performed.

This seems quite fast timing however, it would be interesting to see what that game is exactly doing though. Are you sure you were emulating FIFO correctly and with the right timings (you can test Chaos Engine first level and Double Clutch title screen) ?
Another interesting thing about this game is its SEGA intro. It has at least 3 different intros it uses, the weird thing is why it decides to use them. I noticed if I don't emulate FIFO correctly it gives me an exploding SEGA type logo when on the USA Genesis. Other regions (Japan/PAL) give different effects too. Maybe someone who is interested can RE it. I tried a few different emulators, Fusion, Gens-GX, Regen, and they seem to give different intros among each other too
There is even morethan just 3 :wink:

You can trigger them randomly in those emulators by doing a soft-reset during the Sega intro, it seems to use the same trick as Eternal Champion or X-Men 2, i.e you can read random HV counter value on reset (and maybe also on power-on, as Kega is doing).
Last edited by Eke on Fri Sep 24, 2010 6:14 am, edited 1 time in total.

RetroRalph
Interested
Posts: 16
Joined: Wed Feb 24, 2010 6:13 am
Contact:

Post by RetroRalph » Thu Aug 26, 2010 8:36 am

Well I don't think they could be buffered like the SMS, since there is little information on this (ie double reading or waiting upto max cycles after a write) you wouldn't think it's buffered. They make sure to tell you that in the TMS manual.

However the other thing to consider is my memory timing. I have 18 slots each active line in 320 mode, however I have no real idea where these are placed (I have my best guess based off observable tests). In theory this game may be tightly timed to read right after a write cycle should have happened but didn't (due to incorrect timings). The chance of this is low in my opinion, but I guess it's possible.

And yeah, I no longer have any problems in any game I've tested, including all the hard ones (they are my first tests each build). I will wait and see what my users say after release, I'm sure there are some that still have issues but it's good not being able to see any issues atm. :) I still have to more accurately do the sprite engine in the VDP (to accurately handle display off affecting sprites), unite the 256/320 modes like the real hardware does and allow SMS modes. So still a lot of work to do really.

Eke
Very interested
Posts: 884
Joined: Wed Feb 28, 2007 2:57 pm
Contact:

Post by Eke » Thu Aug 26, 2010 9:00 am

RetroRalph wrote:However the other thing to consider is my memory timing. I have 18 slots each active line in 320 mode, however I have no real idea where these are placed (I have my best guess based off observable tests). In theory this game may be tightly timed to read right after a write cycle should have happened but didn't (due to incorrect timings).
I have seen where CRAM writes were performed by testing a few demos on real hardware. As far as I remember, access slots are spaced each 2-cell, with 3 slots out of 4 being used for CPU access. This is during active display though and the slots are not exactly timed with pixel 0 of active display. At the end of active line (I guess that this is when VDP have parsed the last data for the line and Vcounter is incremented /Hint is triggered), 2 writes are performed at full speed and presumly, some writes are performed during HBLANK at unknown spots.

This would require extended testing on real hardware though, VRAM bus analysis, etc... until then, there is no real point in emulating this if you are not emulating it exactly as it should, it's not more accurate than current emulators, it's just different approximation at different granularity :wink:
I still have to more accurately do the sprite engine in the VDP (to accurately handle display off affecting sprites), unite the 256/320 modes like the real hardware does and allow SMS modes. So still a lot of work to do really.


Some games to test: make sprites jump so that they are outside of the active screen in Sonic 2 (player 2 screen) or Power Instinct / Deadly Moves. First line of sprite should not be rendered (verified on real hardware).
Last edited by Eke on Thu Aug 26, 2010 9:02 am, edited 1 time in total.

RetroRalph
Interested
Posts: 16
Joined: Wed Feb 24, 2010 6:13 am
Contact:

Post by RetroRalph » Thu Aug 26, 2010 9:01 am

On the intro, that would make sense. I just implemented a random line counter for startup (which I guess the real system is roughly doing) and I got a variety of different intros, nice! Why do you only do a random line start on soft reset in Gens-GX ?

Eke
Very interested
Posts: 884
Joined: Wed Feb 28, 2007 2:57 pm
Contact:

Post by Eke » Thu Aug 26, 2010 9:14 am

RetroRalph wrote:On the intro, that would make sense. I just implemented a random line counter for startup (which I guess the real system is roughly doing) and I got a variety of different intros, nice! Why do you only do a random line start on soft reset in Gens-GX ?
Actually, what I do is compute a random cycle start value for CPU on soft reset, because VDP is still running when 68k and Z80 are reseted, and when you hit the RESET button, they could start anywhere in the frame.

I guess that the same thing could happen on hard reset (cold start): as I understood it, /RESET lines could be set high randomly on each startup, because of active components inside the hardware (capacitors ?). this means that theoretically, different chips can start with random timing offset.

I did not bother implementing this for hard reset but if would be easy to do.

PS: it's not called Gens-GX but Genesis Plus GX because it's not a Gens port but I'm sure you were already aware of that :wink:

Eke
Very interested
Posts: 884
Joined: Wed Feb 28, 2007 2:57 pm
Contact:

Post by Eke » Thu Aug 26, 2010 9:33 am

Out of topic, but still regarding Soft-Reset, it's not completely clear what happen exactly: an example with the game Defender in Williams Arcade Classic that was discussed some time ago on another topic. In all emulators (that supports soft-reset off course), if you do a soft reset after this game starts then select it again, the start menu will be corrupted as if VRAM was not setup correctly. I verified this does not happen on real hardware.

Maybe it has something to do with memory refresh during RESET ?

RetroRalph
Interested
Posts: 16
Joined: Wed Feb 24, 2010 6:13 am
Contact:

Post by RetroRalph » Thu Aug 26, 2010 9:56 am

Well someone on SMSPower mentioned the fact there shouldn't be THAT much difference on the reset lines, but it seemed enough on the SMS end to give you different enough R values on the Z80. When it comes to the MD I guess it's a question of where does the VDP start upon power on. Maybe its vcounter is simply filled with random garbage upon power on, it's the only way you could explain how games can use the HV counters for random activity. In a way, it could be designed in this manner so that developers could have something random to use.

And sorry about the name confusion, not sure why I abbreviated it to GensGX when there is actually an emulator called Gens. Silly me.

Eke
Very interested
Posts: 884
Joined: Wed Feb 28, 2007 2:57 pm
Contact:

Post by Eke » Thu Aug 26, 2010 10:13 am

RetroRalph wrote:Well someone on SMSPower mentioned the fact there shouldn't be THAT much difference on the reset lines, but it seemed enough on the SMS end to give you different enough R values on the Z80. When it comes to the MD I guess it's a question of where does the VDP start upon power on. Maybe its vcounter is simply filled with random garbage upon power on, it's the only way you could explain how games can use the HV counters for random activity. In a way, it could be designed in this manner so that developers could have something random to use.
.
When you hit soft-reset button, the VDP is still running and is not reseted, so the CPU could restart anywhere during VDP frame and HV counter reads would return random value.

Now, on power-on, /RESET lines are not cleared at the exact same moment on each startup , so the VDP could perfectly resets its internal state (and HV counter) to a known and fixed state when it is reseted but still, the CPU would start running while the VDP has run a random number of cycles, which explain the random HV counter value on CPU read.

HardWareMan
Very interested
Posts: 745
Joined: Sat Dec 15, 2007 7:49 am
Location: Kazakhstan, Pavlodar

Post by HardWareMan » Thu Aug 26, 2010 11:50 am

There are two reset signals in MD: MRES and VRES. MRES is global reset signal, affected to whole system. VRES clears only small part of system, and do not reset raster schema of VDP. So, there are have an difference to initial hardware state from this signals. And sometimes, when M68K hangs there are not enough VRES - system still not responding. Only power cycle or MRES solve that (got this hangs on UMK3 with MD one clone).

Charles MacDonald
Very interested
Posts: 292
Joined: Sat Apr 21, 2007 1:14 am

Post by Charles MacDonald » Fri Aug 27, 2010 4:49 am

So the problem with Bonkers is that it expects the data in the FIFO will be written to VRAM before the VRAM read is processed?

If you always write to VRAM instantly (no FIFO emulation) does Bonkers work fine?

Are there any games that specifically test for a fifo full or partially full
condition in the status flags? E.g. they'd hang if you kept the flags reset all the time?

I wonder if there are two ways the VDP handles this:

If there were two periodic slots for VRAM access, one for a write, one for a read, then the VDP would unload the first word in the FIFO before doing a write. This means that reads don't wait for the FIFO to be emptied, just that by the time a read occurs you can guarantee the topmost word in the FIFO has been written to VRAM. This is probably bad because the read would screw up the address pointer for the following writes. But it means the 68K is stalled for the least amount of time. Conversely it could do the read first, then the write.

The other way is just one slot where writes are given priority over reads, and in this case the FIFO would always be completely emptied by the time a read would be processed. I guess in theory that means the 68K could be stalled for a long period of time (4 vram writes and 1 vram read) before the instruction that writes the read command to the control port finishes. And this ensures the address pointer is valid.

I think both methods could be equally valid. Or maybe not. :D What do you think?

RetroRalph
Interested
Posts: 16
Joined: Wed Feb 24, 2010 6:13 am
Contact:

Post by RetroRalph » Fri Aug 27, 2010 5:32 am

If you simply write the data then Bonkers is fine. It's only when you have the possibility of the 68K still running and the FIFO is being processed in real time will you hit this bug. As far as I'm aware only RetroCopy works in this manner.

I don't know about having two separate "user" cycles, the SMS VDP is based upon the notion of letting the "user" have a slot here and there during the active line. Giving two slots when most of the time the read would be wasted doesn't seem like the best thing to do. So each write slot would also correspond to a possible read slot.

The engineers would have structured the Genesis VDP based highly off the SMS one. Given that mode 4 (SMS) exists you would think they have some sort of "read buffer" in there, whether it's active for mode 5 or not is the question. The most likely explanation in my opinion is that since FIFO has a known ability to stop the 68K when it's full, that it does it on reads always to ensure you have valid data (waiting for a slot). Given that Sega doesn't even mention anything about reads in the documents I've seen it would imply there is no "special cases" with reading. To have no special cases you have to block the 68K, there is no other way around it really.

Of course this is under the assumption that there are no special cases, maybe there is and Sega never documented it. It also raises questions about the Z80 and what happens when it reads. How much is the "Read buffer" used in mode 5? Does writing affect it? Does the VDP need two slots to read a word or just one?

RetroRalph
Interested
Posts: 16
Joined: Wed Feb 24, 2010 6:13 am
Contact:

Post by RetroRalph » Fri Aug 27, 2010 8:04 am

Ok, so here is how I am currently emulating VDP writes/reads upon some reflection and testing. It may interest some of you, I don't know if it's 100% correct but there are no issues in any game I have tried. RetroCopy's VDP is run at the cycle level, so if my timing is off even slightly there are usually severe bugs, however using what I outline below there are no bugs (that I've noticed), and there are improvements in regards to my CRAM dot emulation (most happen off screen now).

1) Upon a VDP read 68K is locked until the read is completed.
2) A VDP data read adds a command itself to the FIFO buffer. In turn which sets the appropriate status register flags. VRAM reads take two slots, others take one slot, just like writing. There is virtually no difference between a data port write or read on the VDP end besides the fact reads will be blocked.
3) There is an internal 16bit read buffer, it is set when a FIFO read command is taken
4) A write that happens with a read code affects the internal readbuffer, however the 68K will never notice this, only the z80 can.
5) A read that happens with a write code set will write some junk (whatever was last stored in the FIFO buffer) at the VDP address
6) Z80 reading data port returns the internal readbuffer immediately, no waiting, so will return what was last read, however it should still block the 68K until the read is finished
7) A write that has a read code set can prematurely end the 68K halt on read and return a different value than expected.


Things to find out
1) Does setting the code to a write command (0x01, 0x03, 0x05) and then doing a read actually set VRAM,CRAM,VSRAM to some junk value? Can be easily tested
2) Does setting the code to a read command and then doing a write set the internal read buffer? (Z80 read could prove this)
3) Why do some games set different codes (0x02 in Chaos Engine). It could purely be a delaying mechanism.
4) verify that the 68K is locked upon a read. Basically do a write during active line with reg[15] set to 0 and immediately follow it up with a read, if you always get the value written answer is yes.
5) Does a write with a READ code prematurely end the 68K halt?


Charles Macdonald in his VDP document wrote that you couldn't write or read VRAM with the wrong code but maybe the behaviour outlined above exists, unless you were specifically looking for it it would be hard to notice. I will keep with this method for the public release (within a week) unless I find any issues with it.

Nemesis
Very interested
Posts: 791
Joined: Wed Nov 07, 2007 1:09 am
Location: Sydney, Australia

Post by Nemesis » Sat Aug 28, 2010 3:43 am

I know some of the answers about VRAM reads, as I've done a lot of hardware tests on port access and FIFO behaviour on the VDP. Here's what I can say from my testing:
-VRAM reads are moved through the same 4-word FIFO as VRAM writes.
-When a VRAM read is requested via a control port write, the read is pre-cached in the oldest entry in the FIFO. IE, if four words are written to the VRAM, 0x1111, 0x2222, 0x3333, 0x4444, and a VRAM read follows, the data read will use the FIFO buffer entry which previously held the 0x1111 value.
-Note that the FIFO is a ring-buffer, not a stack. If two writes are made to VRAM, the FIFO will be advanced two positions, regardless of whether both writes were held in the FIFO at once, or if the first write was moved out of the FIFO before the second write was made.
-Since a VRAM read uses the oldest entry in the FIFO to cache the read data, a VRAM read can only be setup if the FIFO is empty, otherwise the VDP wouldn't know which slot to retrieve the cached data from if the FIFO was advanced.
-If the FIFO is not empty when a control port write is made to setup a VRAM read operation, I'm currently not sure if the VDP stalls the write to the control port, or if the VDP stalls the read from the control port until the first byte of the read has been pre-cached. Either way, all writes are committed before the read is processed.
-Note that when performing a VRAM read from CRAM or VSRAM, only the valid bits from those memory buffers are set in the FIFO buffer entry being used to cache the read. The previous state of the unused bits retain their previous values. It's possible to use this phenomenon to observe the way the VDP uses the FIFO, since some bits of the FIFO buffer can be read without updating their contents. Here is the 16-bit mask for the valid bits in the CRAM buffer: 0000 1110 1110 1110. And here is the 16-but mask for the valid bits in the VSRAM buffer: 0000 0111 1111 1111.

A couple more questions I can answer:
1) Does setting the code to a write command (0x01, 0x03, 0x05) and then doing a read actually set VRAM,CRAM,VSRAM to some junk value? Can be easily tested
Reads can only be performed from the following command values:
0000b : VRAM read
0100b : VSRAM read
1000b : CRAM read
1100b : *8-bit VRAM read (see notes)
The last setting performs a normal read from VRAM, except that the upper 8 bits are forced to 0. In other words, the lower byte returns the same value as a normal VRAM read, while the upper 8 bits always return 0. Reads from any other targets cause the VDP to completely lock up. Resetting the console from this state has no effect, it needs to be power-cycled to restore the VDP.
2) Does setting the code to a read command and then doing a write set the internal read buffer? (Z80 read could prove this)
The VDP uses the 4-word FIFO to cache read data. Performing a write to an invalid write target (such as a read target) does write the data to the FIFO, but it doesn't get written anywhere. IE, the data is written to the FIFO, but the actual write is ignored.

Nemesis
Very interested
Posts: 791
Joined: Wed Nov 07, 2007 1:09 am
Location: Sydney, Australia

Post by Nemesis » Sat Aug 28, 2010 4:21 am

One point I forgot to mention:
-Successive read operations use the same FIFO entry. Read data is always pre-cached in the oldest entry in the FIFO buffer, and is not advanced after a read is performed.

Oh, as a sidenote, I'm fairly certain that any attempt to write to the control port is held pending until the FIFO is empty. If this wasn't the case, if a write was held in the FIFO, and a control port write was made, that control port write would change the command code and command address registers, which would affect the way the pending writes were processed, and cause them to go to the wrong address or memory target. Either every data port write also stores the address and command code registers at the time it was made (highly unlikely), or writes to the control port are held pending until the FIFO is empty, which would also mean that reads cannot be initiated until the FIFO is empty, since that requires a control port write to setup.

Post Reply