UMDK manufacturing, part 2: Software

[prophet36]

Here's the heartbeat trace file

OK, so here's a technical description of what happened during this boot. Skip to the end if you're not interested.

It looks like the very first instruction of the menu program gets read back incorrectly:

Code: Select all

$ logread trace.log | grep -E '( 420200 | 420202 )'
            16690453 C WB 420200 46FC
            16690592 C WB 420202 2700
            21345059 C RD 420200 1B25
            21345084 C RD 420202 2700
$

Those two words 0x46FC and 0x2700 written at 0x420200 and 0x420202 respectively are supposed to make up a single instruction ("move.w #0x2700, sr"), to initialise interrupts and switch the CPU from supervisor to user mode. But they're read back as 0x1B25 and 0x2700, which is interpreted by the CPU as two separate instructions:

Code: Select all

$ printf '\x1B\x25\x27\x00' > boot.bin
$ dis68 boot.bin 0 2
0x000000  move.b -(a5), -(a5)
0x000002  move.l d0, -(a3)
$

In this particular run, a3 happened to come out of reset containing an odd address, so the second instruction causes an address error:

Code: Select all

21345059 C RD 420200 1B25
21345084 C RD 420202 2700
21345147 C RD 420204 4AB9
21345181 C WL FFFFFC FFFF
21345198 C RD 420206 00A1
21345307 C WB FFFFFE 0206
21345332 C WB FFFFFA 2700
21345358 C WB FFFFFC 0042
21345383 C WB FFFFF8 2700
21345408 C WB FFFFF6 FFFD
21345433 C WB FFFFF2 2705
21345459 C WB FFFFF4 FFFF
21345476 C RD 00000C 00A1
21345501 C RD 00000E 0364
21349597 HEARTBEAT
21353693 HEARTBEAT
         :

...and despite the fact that the address-error vector had been earlier initialised correctly to 0x420364:

Code: Select all

$ logread trace.log | grep -E '( 48000C | 48000E )'
            16654560 C WB 48000C 0042
            16654699 C WB 48000E 0364
$

...it actually gets incorrectly read back as 0xA10364. But there is no automatic DTACK generation at that address, so when the CPU jumps there it just hangs, forever.

The most likely cause of this is an observation I made since adding the heartbeat-tracing: the CPU takes a LONG time to start fetching instructions from the cart, after reset is deasserted: on my MD it's about 120ms, but on yours it's closer 350ms. Both of these are several orders of magnitude longer than it would be on a "vanilla" 68000 system, so I'm guessing the MegaDrive has some of its own initialisation to do before starting the CPU.

Unfortunately, because of the way UMDK interleaves SDRAM refresh cycles with regular reads and writes, this means the SDRAM will not be getting any refresh cycles for several hundred milliseconds after reset, which means the SDRAM will be rather unreliable during the boot process, hence this crash.

I have an idea for how to fix it, by doing continuous-refresh (which I normally do only during reset) right up to the point where the CPU starts doing fetches, but it will take some time to think about how to do that safely.

[Montserrat]

The most likely cause of this is an observation I made since adding the heartbeat-tracing: the CPU takes a LONG time to start fetching instructions from the cart, after reset is deasserted: on my MD it's about 120ms, but on yours it's closer 350ms.

Mmm diferent boards have diferent 68k processors, even some of them are not motorola, may be there are different behaviour between revisions:

For the earlier/later MD1 models we have: Signetics SCN68000C8N64 // Motorola MC68000P8

For the earlier/later MD2 models we have: Motorola MC68HC000FN8 // Sega 315-5960 ASIC (this is an ALL-IN ONE 68k, VDP,Z80 video encoder chip and more.)

You probably have Motorola MC68HC000FN8, and the rest of us probably Motorola MC68000P8, the nomad has a all-in one revision. May be theres some diferences (apart from clock speeds) in how , and how fast rom is accesed. If you think it can be usesful i can do some research on the service manuals and schematics.

[prophet36]

Yes, those differences could easily be significant. Or it could be something simple like different pull-up resistor values on the cartridge slot's reset signal. Either way, it's just a fact of life that there is a big delay on startup, which I will have to deal with, one way or another.

[prophet36]

OK Montserrat, can you try this updated FPGA binary?

Code: Select all

cd ${HOME}/umdkv2-bin
mv fpga.bin fpga.bin.20151212
wget -q https://dl.dropboxusercontent.com/u/80983693/umdkv2/fpga.bin
flcli -v 1d50:602b -p J:A7A0A3A1:spi-talk.xsvf
gordon -v 1d50:602b -t indirect:1 -w fpga.bin:0

This just does back-to-back SDRAM refresh cycles until the 68000 starts executing.

[Montserrat]

edited. Continue reading.

[TestSubject06]

Just figured I'd poke my head in here and say that I'm following this very closely. The PCBs will arrive for me in the next week or two, as will all of the components. So before too much longer I'll be able to join in on the debugging fun.

Which also means I'll have some spare PCBs to send people that want to get in on this.

[prophet36]

I'm following this very closely

That's great, the more the merrier. We're actually in a much better position now than we were even a few weeks ago. There's more clarity on which of the observed problems people are seeing are due to bad design on my part, and which are due to manufacturing errors. And when something does go wrong, we have some quite powerful tools to help us figure out exactly what's happening.

If you do have spare PCBs, do you intend to solder multiple boards and sell the spares?

[Montserrat]

OK Montserrat, can you try this updated FPGA binary?
This just does back-to-back SDRAM refresh cycles until the 68000 starts executing.

I've tried the new firmware, and it works perfect! Now its booting standalone in a second, it loads roms, can use DDD, even from nomad. Awesome job.

Noticed one more thing. Seems that it cannot save games. Those games that support SRAM save do not preserve any data once powered off. I am doing something wrong? it is supossed to save the data at the SD?

[prophet36]

Noticed one more thing. Seems that it cannot save games

I discussed this here. Having reacquainted myself with the code these last couple of weeks, I can confirm that the reason game-saves don't work is because I explicitly disabled writes to the 0x000000-0x3FFFFF region. Getting game-saves to work so you can save them over USB would be fairly easy, but writing to the SD-card would be quite difficult - the menu program currently can only read the FAT32 filesystem on the SD-card; supporting writes to SD-card will be more difficult.

[TestSubject06]

I'm following this very closely

If you do have spare PCBs, do you intend to solder multiple boards and sell the spares?

That all depends on how my board comes out. I've done a little bit of drag soldering, and I've done some SMD resistors and caps, but some of those smaller 3-pronged components are going to give me hell, since I have nothing to hold them down with.

If I hate the quality of my boards I might sell them as incomplete sets to make back a little of what it's cost to get all the components + the PCBs (Over $230...)

[Montserrat]

Noticed one more thing. Seems that it cannot save games

I discussed this here. Having reacquainted myself with the code these last couple of weeks, I can confirm that the reason game-saves don't work is because I explicitly disabled writes to the 0x000000-0x3FFFFF region. Getting game-saves to work so you can save them over USB would be fairly easy, but writing to the SD-card would be quite difficult - the menu program currently can only read the FAT32 filesystem on the SD-card; supporting writes to SD-card will be more difficult.

Ouch...i assumed UMDK was able to do that :S. is there any chance it will be posible in the future?

[Burbruee]

Ok, I tried the new files and now my umdk also boots to the menu program without workaround method. Great work prophet!

I actually thought I had a crash happen with the sd-card inserted, because it showed the menu program text for a split second and then went to a black screen. But actually I didn't realize we do not support folders in the menu program and I had organized the root of the card with folders only, for regions, homebrew, dev-folders and such. I put some roms directly in the root and I was finally able to see some files.

Just as with USB-loading I am however experiencing crashes in-game on certain games while many others work flawlessly. I will dig up that macro lens this coming week and have a closer look at the board. But it's strange to me that some games can run for 40+ hours without crashing and other games crash after 10 seconds.

[prophet36]

Ouch...i assumed UMDK was able to do [game saves]. is there any chance it will be posible in the future?

It's certainly possible, but I have no immediate plans to do it. OTOH, it's an open-source project, so everyone is free to implement new features, as long as they adhere to the terms of the licence agreement.

But it's strange to me that some games can run for 40+ hours without crashing and other games crash after 10 seconds.

Yes, it is strange. Did you try what I suggested with a needle and a magnifying-glass on the /OE signal?

[Montserrat]

At this moment only 3 of 5 umdk has been tested, and only 1 of that 3 is fully working as intended (mine).

I think the manufacturing process is somewhat ok but it caused some problems. Is there any way to do hardware checks, via software or via hardware? (db-electronics guy said something). May be if i ask the manufacturer to do some basic checklist before ship, then will be more safer.

Do i release the manufacturer to public or may be we try another 5x run for more testers?

[MintyTheCat]

At this moment only 3 of 5 umdk has been tested, and only 1 of that 3 is fully working as intended (mine).

I think the manufacturing process is somewhat ok but it caused some problems. Is there any way to do hardware checks, via software or via hardware? (db-electronics guy said something). May be if i ask the manufacturer to do some basic checklist before ship, then will be more safer.

Do i release the manufacturer to public or may be we try another 5x run for more testers?

Usually something known as a 'Test-Jig' is used to test built hardware. It subjects the hardware to known patterns and checks the responses.
Generally it is added on to a developed project and catches many issues early.

https://en.wikipedia.org/wiki/Jig_(tool)

db was making something that could be used to get inside and test the hardware so it would serve the same purpose.

But yes, if the manufacturer had that it would make the process a lot easier.

It may be better to wait for db to get his UMDK manufacture worked out as he has all the necessary tools at his place pretty much - which is why he was asked to handle the manufacture as we had foreseen lots of issues due to soldering these tiny components.
Essentially, we want them to be manufactured using a machine that solders the components to the PCBs for us without any need for hand soldering and then to have a means to test the resultant PCBs once manufactured.

As I recall we got up to 30 orders for UMDK kits so it is just enough to justify getting the process automated but in truth it is a more specialised piece of hardware that the masses would not be looking at getting hold of and as such more expensive to build small batches of.