Exodus 2.0 + Open Source Release

Official support forum for the Exodus Emulation Platform

Moderator: Nemesis

Nemesis
Very interested
Posts: 791
Joined: Wed Nov 07, 2007 1:09 am
Location: Sydney, Australia

Post by Nemesis » Tue May 05, 2015 12:35 am

Dr. MefistO wrote:Nemesis, I'm using PVS Studio to detect that bugs, that I've commited. You can ask me for it at PM. Then you will see what my every change means.
I'm familiar with PVS Studio, I've run both PVS Studio and Cppcheck over the Exodus codebase in the past. They can be a very useful tool and help to identify errors. I have no problems with using them, and I can see you're not doing it blindly. My problem is more about the fork and commit process, which I'll cover a bit below.
I know that I've comitted too many items, but what I have to do if it really valuable changes?)
You need to treat a fork as a branch. A branch should be a staging area for a single task or job. Fixing the .rc include references is a single job. That should have its own fork, be reviewed, and merged to the main repository separately. Your pass over the code with CodeMaid should be a separate fork which receives a separate review, and in that case, I can tell you that I would be rejecting the CodeMaid changes. I would accept the .rc changes though. By bundling them together in the one fork, I can't do that, so I have to reject the whole thing.

The commit history of the repository is just as important as the current state of the repository. Changes need to be compartmentalized and tracked properly. You can work on more than one task at a time, but the changes need to be isolated from each other until they're complete, and ready for merging to the "mainline", at which point they need to be able to be merged separately. I'll create a ticket in the Jira system for your fork when I'm merging it, and use that ticket number in the merge commit, so that there's some proper historical tracking of the change, and a write-up on where it comes from and why it was done.

I've got a lot of experience with this kind of thing, and believe me, mega-changes are bad news, and cause big headaches. When bugs are introduced by changes, which they inevitably are, it needs to be easy to trace back through the commit history for a set of files, or the repository as a whole, and observe the relevant changes that have occurred, separating each job from each other and testing as you go to narrow down the changeset that introduced the problem. If you have a mega-change that re-formats all the files in the codebase, it introduces massive "white noise" into the diffs, which makes narrowing down the cause of the issue a nightmare, especially when that code change mixes reformatting with functional changes. If a change is formatting only, you know you can totally discount it, and do your diffs "around it", but when functional changes are mixed in, it becomes a pain point for years to come, and that commit will become the "usual suspect" for introduced errors. You've also created a bit of a merging nightmare for yourself, because I'm making my own code changes independently on files you've modified, and what's worse, the code formatting change you tried to do conflicts with my changes, so if you wanted to maintain your formatting changes, you're going to have to edit my changes and format them too in order to merge.

Sometimes large single changes are necessary, but they need to be coordinated and carefully managed, and they need to be done for a good reason.
CodeMaid is not so bad, as you think: code is more readable, comments are still there, and so on..) And this is for uniformity of your code and code by others. I don't know how to explain. But to develope code by few people in one style it should be code in one style.=)
The code currently is uniform, CodeMaid is just imposing a different style entirely. The etiquette for dealing with these kind of style issues is simple: "When in Rome....". Different areas of code can use different code styling, but files and libraries should be consistent within themselves. People making changes in a source file with 2000 lines should follow the formatting styles of the other 2000 lines, not write 5 lines using a different convention. That's the kind of thing I'd manage at the merge step, so I'd be keeping style issues in check there anyway. I wouldn't impose a single set of requirements on entirely new plugins or components though. As long as they're consistent in themselves, it doesn't cause a problem. This is C++ we're talking about, there are 1000 different styles, and no accepted standard formatting conventions.

That the new code is "more readable" is also completely subjective. You may believe the code has become more readable for you, but there's no way you've looked at every file you just changed, and people will disagree on what's more readable even in the simple cases. I also know from experience that for every formatting convention, there's the "exception to the rule" cases, most especially with comments, and code formatting tools don't understand that. I looked at a bunch of files and cases in that commit that I knew CodeMaid would screw up, and it has. Lots of comments are using fixed layout and spacing. Static lookup tables are similarly being laid out using deliberate spacing to align columns and make them easier to view, and errors easier to spot. CodeMaid is wrecking these.

You don't change convention across a codebase on a whim without careful thought and review, and especially not sandwiched between several other jobs in a fork. If your fork introduces an error, and I want to do a diff to find the point where it was introduced, it'll be useless, because CodeMaid has changed half the lines of code in every file in the repository. There needs to be a very good reason to reformat code like this, and if it is to be done at all, it needs to be carefully planned, tested to exhaustion before it's done "for real", and when it is done for real, it needs to be done quickly, merged quickly, and coordinated with everyone who may be working with that codebase.
I will update my pull request with some new commits, and you can check them as accurately as you want.
The CodeMaid changes will not be merged. Since that was the second commit, and all your changes after that are based on the reformatted code, your fork is poisoned. I'd suggest you create clean forks, one for each logical set of changes. Your .rc changes should be in one, the bugfixes in another, and your performance changes in another. Both the bugfix and performance changes you've made have some errors that need correcting, so once they're isolated into their own forks, I can review those properly and discuss them with you.


I hope I don't put you off making changes with my comments. I do appreciate your efforts to make improvements, we'll just need to take things a little slow here and get the hang of the fork, review, and merge process, and discuss any large-scale changes that touch a lot of files before the changes are made, because there might be other changes in development, or future plans, that would affect that job. I've also never done this on bitbucket before, so learning their system is a task in itself.

If you can start with a clean fork for just the .rc file changes, I'll try adn get the CLA up so that I can actually accept your changes, and we'll try and go through the review and merge process on that fork. With the first fork completed and merged, it'll clarify the process for me too, and I should be able to write a document on the process to help other people too.

Nemesis
Very interested
Posts: 791
Joined: Wed Nov 07, 2007 1:09 am
Location: Sydney, Australia

Post by Nemesis » Tue May 05, 2015 12:44 am

MetalliC wrote:Nemesis
congrats with release, great work!
Thanks :)
does YM in Exodus include all latest findings ? I've heard there was researches/tests about envelopeCycleCounter, and it shows its not such simple, like decrementing starting from 4095, and reload to 4095 at 0, but I dont see this in Exodus code
No, the YM2612 core isn't perfect, there are a couple of known things in there that need to be done. The busy flag delays, test register, operator "staging" register (as per MAME), DAC input precision, etc. I haven't done much work on it in the last couple of years, because I was working on getting the cycle-accurate VDP rendering process complete. There's also a lot of notes in there about things that need to be tested on the hardware. I've tried to note any point of behaviour where I haven't confirmed a query I had about how the hardware actually behaved in a given situation. When I did the YM2612 reverse engineering process, I didn't finish testing everything, I simply moved on to other areas, and in a lot of cases, I thought of more tests to perform later. If you can see any aspects that seem to be incomplete or inaccurate, I'd welcome contributions to fix it. Please note how extensive the commenting is in the YM2612 core though. Because there's so little documentation that describes how this device works, the comments really need to be extensive, so any change should be very extensive in explaining why things are done in a particular way, and should explicitly state that it has been tested on the hardware if it has been. When I'm creating a Jira ticket for the job, I'll also want something to explain how the new behaviour is known to be correct too, such as a reference to a forum discussion, example program, or situation within a game where the behaviour can be observed. This will allow the results to be re-evaluated later if newer research disputes a previous finding or implementation. This may seem overly particular, but there needs to be a high degree of confidence around changes to these cores, or where there is uncertainty, it needs to be noted, because the learning barrier is very high, and the testing process can be very difficult. If something is added that is later thought to be wrong, and there's not enough information to show why it was thought to be correct at the time, it'll be very hard to have confidence that the new change isn't missing or forgetting something that was understood previously.
also I've noticed "Mr.Nutz Hoppin' Mad" won't work, both original and "fixed" versions hang at very start, sometimes after showing Sega logo.
Shadow wrote:This night was very boring, so I decided to check the games.

Bram stoker's dracula: freezes at the beginning of the first level.

Killing game show: Black screen.

Mega-lo-Mania: Black screen.

Speedy Gonzales - Cheeze Cat-astrophe: no sound.

Strider II: no collision with enemies, immortality.

Shadow of the Beast: no collision with enemies.

The Lawnmower Man: flashing vertical stripes on 3d levels.
Thanks for the reports! I should probably start keeping a list of games with known issues, that way I can investigate each one, and note the cause of the problem. It'll also greatly assist in testing when I believe I've corrected the problem that caused the issue.

Mask of Destiny
Very interested
Posts: 615
Joined: Thu Nov 30, 2006 6:30 am

Post by Mask of Destiny » Tue May 05, 2015 1:22 am

Congrats on the release Nemesis!

Strider II gave me a little trouble as well. It's one of the few games that uses the sprite collision flag and there will indeed be no collision in that game if the flag is not implemented. It also uses the BCD instructions for the timer so if Shadow means time never runs out when he says "immortality", then that's a likely culprit.

Nemesis
Very interested
Posts: 791
Joined: Wed Nov 07, 2007 1:09 am
Location: Sydney, Australia

Post by Nemesis » Tue May 05, 2015 1:40 am

Mask of Destiny wrote:Congrats on the release Nemesis!
Thanks, it's been a long time coming!
Strider II gave me a little trouble as well. It's one of the few games that uses the sprite collision flag and there will indeed be no collision in that game if the flag is not implemented.
Yep, figured as much. I haven't implemented the sprite collision flag yet, because I haven't finished getting cycle-exact measurements from the VDP for the timing yet. I know the sprite collision flag is set during the sprite pattern reads in hblank, but I don't know the cycle delay between the read of the pattern row and the setting of the flag for each pixel. I haven't wanted to implement it until I know it's perfect, because if I get the timing slightly wrong, chances are it'll never be fixed.
It also uses the BCD instructions for the timer so if Shadow means time never runs out when he says "immortality", then that's a likely culprit.
Thanks, I'll check that out. I'd be surprised if the BCD instructions had a problem after all this time, but it's certainly possible. I found the score counter in "Columns" was a good test for that too, I had bugs in my BCD implementation at first that broke that. I remember rotating in Ecco was a good test for my DIV opcodes, lots of bugs there at first. I was amazed really in the early days just how badly things could be broken, with most games still running seemingly fine. Goes to show how few games actually make use of more than a small set of features or instructions.

Dr. MefistO
Interested
Posts: 47
Joined: Wed Jan 08, 2014 3:39 pm

Post by Dr. MefistO » Tue May 05, 2015 6:41 am

Nemesis, you're right!
So, I decided to redo my work with your sources but without applying of CodeMaid.

Could you reject all my commits after first one?)

P.S. How can I create a branch on your repo?

Eke
Very interested
Posts: 884
Joined: Wed Feb 28, 2007 2:57 pm
Contact:

Post by Eke » Tue May 05, 2015 8:27 am

Killing game show: Black screen
Bram stoker's dracula: freezes at the beginning of the first level.
Mega-lo-Mania: Black screen.
Not exactly sure for the first one (I only remember it was sensitive to how interrupts are handled) but the two others definitively rely on reading the VINT flag in VDP status before VINT actually occurs. This is one of the edge case you are already aware and would require cycle-accurate (or at least bus cycle) 68k emulation.

On that topic, I might be able to help you. At my work place, I actually have access to a logic analyzer with a specific 68k module which traces code running on a 68000 target including all bus access, exceptions, etc.

The trace format is shown below:

Code: Select all

     3380	1   	006FF4 	2039  	  MOVE.L   03404C,D0      	7   	                250.000 ns
     3381	1   	006FF6 	0003  	  ( EXTENSION )           	7   	                250.500 ns
     3382	1   	006FF8 	404C  	  ( EXTENSION )           	7   	                249.500 ns
     3383	1   	006FFA 	207C  	  MOVEA.L  #0003C445,A0   	7   	                250.500 ns
     3384	1   	03404C 	2014  	  ( READ )                	7   	                625.000 ns
     3385	1   	03404E 	00BF  	  ( READ )                	7   	                500.000 ns
     3386	1   	006FFC 	0003  	  ( EXTENSION )           	7   	                250.000 ns
     3387	1   	006FFE 	C445  	  ( EXTENSION )           	7   	                250.000 ns
     3388	1   	007000 	4EB9  	  JSR      006ED0         	7   	                250.000 ns
     3389	1   	007002 	0000  	  ( EXTENSION )           	7   	                250.000 ns
     3390	1   	007004 	6ED0  	  ( EXTENSION )           	7   	                250.000 ns
     3391	1   	006ED0 	E388  	  LSR.L    #1,D0          	7   	                250.000 ns
     3392	1   	0200AC 	0000  	  ( WRITE )               	7   	                250.000 ns
     3393	1   	0200AE 	7006  	  ( WRITE )               	7   	                250.000 ns
     3394	1   	006ED2 	55D8  	  SCS.B    (A0)+          	7   	                250.000 ns
     3395	1   	006ED4 	4E71  	  NOP                     	7   	                250.000 ns
     3396	1   	03C445 	--00  	  ( READ )                	7   	              1.000,000 us
     3397	1   	006ED6 	5BD8  	  SMI.B    (A0)+          	7   	                250.000 ns
     3398	1   	03C445 	--00  	  ( WRITE )               	7   	                687.500 ns
     3399	1   	006ED8 	E588  	  LSR.L    #2,D0          	7   	                250.000 ns
     3400	1   	03C446 	00--  	  ( READ )                	7   	                625.000 ns
     3401	1   	006EDA 	55D8  	  SCS.B    (A0)+          	7   	                250.000 ns
     3402	1   	03C446 	00--  	  ( WRITE )               	7   	                687.500 ns
     3403	1   	006EDC 	4E71  	  NOP                     	7   	                250.000 ns

The interesting thing is that it shows both the disassembly and the exact order of prefetch and R/W cycles so it can help to figure more easily the exact timing of R/W bus cycles for any instructions.

I know you already have access to logic analyzer but I figured that this tool might be helpful since everything is already done and configured to produce an exploitable trace. All it needs is some basic code I can easily upload and run to test all instructions with each possible addressing mode accessing external bus.

Shadow
Very interested
Posts: 257
Joined: Wed Sep 16, 2009 7:13 am
Location: Russian Federation

Post by Shadow » Tue May 05, 2015 4:51 pm

Nemesis wrote:Thanks for the reports! I should probably start keeping a list of games with known issues, that way I can investigate each one, and note the cause of the problem. It'll also greatly assist in testing when I believe I've corrected the problem that caused the issue.
:)

Another World: minor problems with control on the title screen (Start \ Password)

Barkley Shut Up and Jam 2: shaking vertical stripes on the character select screen.

Barkley Shut Up and Jam: goes into a loop of resets on SEGA logo. (probably SRAM\EEPROM issue)

Battlemaster: run slowly (incorrect clearing of Z80 registers on a reset) Fixed in 8b374cd build.

Bugs Bunny in Double Trouble: hangs on the title screen.

Chess [Unl]: run slowly (incorrect clearing of Z80 registers on a reset) Fixed in 8b374cd build.

Decap Attack: run slowly (incorrect clearing of Z80 registers on a reset) Fixed in 8b374cd build.

Metal Sonic HYPERDRIVE: run slowly at title screen (48-60fps) and gameplay in fullspeed (incorrect clearing of Z80 registers on a reset?) Not fixed.

Talmit's Adventure: hangs in pause mode

Nemesis
Very interested
Posts: 791
Joined: Wed Nov 07, 2007 1:09 am
Location: Sydney, Australia

Post by Nemesis » Wed May 06, 2015 12:15 am

tryphon wrote:
Nemesis wrote:Seems there's scope for another debug window here. How about a window which shows the output image, but when you move the mouse over the window, it shows a floating window that tells you the active scan pixel index, palette row and index, source layer, and pattern/mapping entry that was used to determine that value? That'd give you not only a way to determine pixel position, but effectively a way to reverse the rendering pipeline and determine what source data was used to produce it. The VRAM pattern viewer uses floating window similar to what I'm thinking of.
I was precisely dreaming of such a window (no kidding) :D :D :D
How about this:
Image
It even identifies pixels that are generated from CRAM writes during active scan:
Image

This popup window will now appear on any VDP image windows if the "Show Pixel Info" VDP image debug setting is enabled. I had to make some changes to the VDP render process in order to ensure all the necessary data was cached at the various points so that the render pipeline could be completely reversed, but it's all in there now. All the data you see on that popup window is cached during the actual rendering process, so it is guaranteed to be totally correct for the image you see on the screen, even when layer removal is being used, or data/registers are being changed during the rendering process. The "HCounter" and "VCounter" settings will give you what you needed in your example, which was a way to determine the pixel size of something, while also giving a way to correlate the timing of the render process with the timing of port access to the VDP.

I've committed these changes into the repository, and they'll be included in the next release. If you want them right now, you can do a build yourself from the source by following the instructions here:
http://www.exodusemulator.com/index.php ... ing-source

Nemesis
Very interested
Posts: 791
Joined: Wed Nov 07, 2007 1:09 am
Location: Sydney, Australia

Post by Nemesis » Wed May 06, 2015 12:33 am

Dr. MefistO wrote:Nemesis, you're right!
So, I decided to redo my work with your sources but without applying of CodeMaid.

Could you reject all my commits after first one?)

P.S. How can I create a branch on your repo?
No worries, let me know when you've got a fork ready for merging. I don't believe there's any way for me to reject individual commits in a fork, it's an all-or-nothing process to take the entire fork, or reject it.

As for branches, there's no way for you to directly create a branch on the main repository, but if you think of a fork as being like a branch, because for all intensive purposes it basically is, then you just follow the same kind of process with your fork as if it was a branch.
Shadow wrote: ....
Thanks for all the testing you're doing! I've created a thread now for reporting compatibility issues: viewtopic.php?t=2029
I'll investigate each one and add it to the Jira issue tracking system so I can ensure they all get resolved.

Nemesis
Very interested
Posts: 791
Joined: Wed Nov 07, 2007 1:09 am
Location: Sydney, Australia

Post by Nemesis » Wed May 06, 2015 12:55 am

Eke wrote:On that topic, I might be able to help you. At my work place, I actually have access to a logic analyzer with a specific 68k module which traces code running on a 68000 target including all bus access, exceptions, etc.

The trace format is shown below:

....
Oh hell yes, that's exactly the kind of output I want. The UMDK is apparently capable of capturing a similar trace log, and that was the tool I was planning to use, but mine is still not finished yet, so I've just been in waiting mode before I could even start on this.

If I write a test ROM to exercise the entire instruction set, could you run it for me and provide the output? I'll just need to know the maximum ROM size, and a usable RAM range that's available on whatever hardware platform you've got there.

I'll try and make this ROM a bit of a regression tester for functionality too, so I'll run batches of tests with particular instructions to make sure they're being evaluated correctly. With this kind of bus-level trace log, I can just push out register data to a memory address, and continually overwrite it with each new result. The trace log itself will capture the data, so it'll make it easy to generate a single mega-log that captures bus timing and result info for the entire instruction set. An M68000 core could then just be wired to output a similar log during its execution, and diff the results with the file from the real hardware, to check timing and logic of all the instructions in a single operation. That'll be beyond awesome.


I know this is a long shot, but you don't have a similar bus logging capacity for the Z80 by any chance do you? I need to get the same level of info for that CPU at some point, but the M68000 is the most critical one.

Eke
Very interested
Posts: 884
Joined: Wed Feb 28, 2007 2:57 pm
Contact:

Post by Eke » Wed May 06, 2015 8:34 am

Nemesis wrote: If I write a test ROM to exercise the entire instruction set, could you run it for me and provide the output? I'll just need to know the maximum ROM size, and a usable RAM range that's available on whatever hardware platform you've got there.
I will not be able to run a standalone test ROM on the target hardware for various reasons but if you provide me assembly code, I am able to patch it in the existing initialization sequence, build a test executable, upload it to target and trace the patched code (the logic analyzer is able to trigger on specific fetched code or memory access address / data).

The code must be off course compatible with the assembler we are using but there is nothing special, as you can see below in this (imaginary) example:

Code: Select all

         move.l    #4,d1 
         clr.l     ramstart
         movea.l   #ramstart,a0 
label_1:
         rol.l     #8,d0 
         move.w    (a0)+,d0 
         move.l    d0,ramstart
         move.w    (a0),d0 
         move.b    #@FF,ramstart+4
         cmpi.l    #constval,ramstart
         dbf       d1,label_1
As for the RAM usable range, I suggest you use symbolic data like in the code above and I will map them myself to proper RAM address.

I'll try and make this ROM a bit of a regression tester for functionality too, so I'll run batches of tests with particular instructions to make sure they're being evaluated correctly. With this kind of bus-level trace log, I can just push out register data to a memory address, and continually overwrite it with each new result. The trace log itself will capture the data, so it'll make it easy to generate a single mega-log that captures bus timing and result info for the entire instruction set.
Yes, that's what I have in mind too. Only limitation is the size of the trace which is limited to 64K entries in one shot. If the code is too large, I will need to trigger at some further code location and make multiple traces.
An M68000 core could then just be wired to output a similar log during its execution, and diff the results with the file from the real hardware, to check timing and logic of all the instructions in a single operation. That'll be beyond awesome.
Indeed. The trace log is displaying columns for the various infos (sample number, address value, data value, mnemonic, interrupt level, and timestamp) but it's easy to either remove the unwanted ones or filter them later in a spreadsheet application if you want to compare with a trace log from another source.

The timestamp in particular gives interesting infos on the number of elapsed cycles but is obviously based on the installed clock (16Mhz here) and depends on existing wait-states in hardware implementation: note that the value can either be relative to the previous sample or absolute.
I know this is a long shot, but you don't have a similar bus logging capacity for the Z80 by any chance do you? I need to get the same level of info for that CPU at some point, but the M68000 is the most critical one.
No, I do not have access to any Z80 target. I admit I was already quite surprised when I first arrived here to see that 68000s were being used :wink:

It seems Z80 microcode is easier to figure though since it does not rely on prefetch like 68000.

tryphon
Very interested
Posts: 316
Joined: Sat Aug 17, 2013 9:38 pm
Location: France

Post by tryphon » Wed May 06, 2015 12:22 pm

Nemesis, I love you !

MrTamk1s
Very interested
Posts: 75
Joined: Sun Jan 04, 2015 10:27 pm
Location: Pennsylvania
Contact:

Post by MrTamk1s » Thu May 07, 2015 1:24 am

Very nice new version and improvements on the debugging and VDP emulation; the emulator finally runs at a decent, usable speed (45-60 FPS) on my Intel i3-core machine (with GameBooster3 enabled and with less overheating of the machine). And just at the right date for a release, with myself resuming work on a homebrew title and just after completing a Computer Architecture uni class :D!

Good job!
SGDK homebrew dev and Unity3D Indie dev.
Sega does what Nintendont!

HardWareMan
Very interested
Posts: 745
Joined: Sat Dec 15, 2007 7:49 am
Location: Kazakhstan, Pavlodar

Post by HardWareMan » Thu May 07, 2015 3:17 am

And what about VDP layer control and sound channel control? Individual enable/disable checkbox will be nice.

Nemesis
Very interested
Posts: 791
Joined: Wed Nov 07, 2007 1:09 am
Location: Sydney, Australia

Post by Nemesis » Thu May 07, 2015 4:30 am

HardWareMan wrote:And what about VDP layer control and sound channel control? Individual enable/disable checkbox will be nice.
VDP layer control is in there, with individual control over high and low priority tiles. Check under the "Graphics Debugger" tab on the standard "Mega Drive Debugger" workspace, or under "Debug -> Mega Drive -> VDP -> Layer Visibility" on the main menu.

As for sound channel control, I'll be adding dedicated windows to visualize and alter the sound output later, but you can disable individual channels right now, you've just got to do it at the register level. Check under the "Audio Debugger" tab. On the YM2612 Debugger, you can "lock" the key on/off state for each channel/operator by a ctrl+click on the checkboxes for the key on/off state. Locking a register will stop programmatic changes from within the emulated environment taking effect, so it'll stay in whatever state you force it to through the debugger. You can do a similar thing for the PSG by locking the volume register state for each channel to a desired value. Again, I'll be adding better debug control over this later, where you actually see a live waveform from each channel/operator output.

Post Reply