MegaLD Dumping Project

[Nemesis]

Nothing you do could possibly drag this out more than I have. How many years has it been now?

I should probably give an update, such as it is. I currently keep on finding a couple of weeks of intermittent time to work on this, then it goes on the backburner again for six months. That's where I'm stuck again right now, but in my last "burst" two months back, I got full digital data+audio ripping, with intact TOC, lead-in, and lead-out, with full subcode data, working in an automated fashion. I need to use a multi-pass approach, takes around 12 hours per disc, per side, but the digital process is proven and in the bag. I've even got multi-pass error correction and subcode verification written, so I can use multiple rips and multiple disks to error correct. That all works a treat, and I'm ready to mass-rip all the digital data from all the disks, but I got stuck again when looking at doing a clean video rip, still trying to get a good solution to get the codes buried in the vblank region. I was looking at doing a software decoding approach, based on the info here:
http://forum.lddb.com/viewtopic.php?f=32&t=2671
I picked up a compatible card, hooked the approproate raw RF point on my player, and was about to build the linux drivers and get a test rip done, but then ran out of time to work on it again. That's where I'm up to, most likely won't have a chance to look at it again until March at the earliest.

[bakudandan]

No need to apologize, these things take time. I'm just glad to read an update.

[Huge]

Possibly related.
https://www.youtube.com/watch?v=klK4UZ5nlqs

[dizzzy]

So what happened here, did the guy who "borrowed" all these games for years do nothing with them?

[Nemesis]

I've never done a youtube video before, but that's probably the best way to sum up where things are right now, as I can show visually what's been done, the hardware involved, and what's still pending and why it's important. To try and give a 30 second summary, here's basically where things are:

1. I can rip the full 2448 bytes of effective sector data for the entire digital content of the MegaLD disks. This includes the lead-in (with the TOC), pre-gap, actual track data, and lead-out, with full subcodes. This took reverse engineering the MegaLD hardware, learning how to cartridge-boot the MegaCD platform in general (never done before), writing custom dumping software to drive it, developing read routines for full subcode data for the MegaCD (never done before), hardware mods to the MegaLD PAC in order to get at the full TOC data, developing a high-speed USB transfer option from the Mega Drive controller port to stream data to the PC, and then writing a chain of half a dozen programs to error correct it, reconcile subcode errors between multiple media sources and successive rips, and merge it all together to get a good, complete data dump. This process has been fully working for around 2 years IIRC.

2. Analog video is still a bit stuck. Originally, I believed I could use off the shelf commercial video capture hardware. This was incorrect. Nothing existed that would properly preserve the important control codes in the vblank region in full. I investigated making my own hardware capture device, and made a few attempts at this, but the data transfer requirements were a killer in terms of bandwidth, and I didn't have enough knowledge on analog signal processing to do a good enough job at it. I was still exploring this space though, and was having a good look at SDR (Software-defined radio) hardware to solve it, until the emergence of the Domesday Duplicator project (https://www.domesday86.com). I've been following this for years, and I've built every hardware revision of it as it came out. I can now get full captures of the raw RF stream from the laserdisc. The remaining problem is the software decoding space. You'd think in 2019, software decoding NTSC video would be a solved problem. It isn't, and until ld-decode there was basically nothing out there that reasonably attempted it. The ld-decode software isn't far enough along yet to decode these disks though. The comb filtering in particular makes a lot of assumptions that don't hold true for the video on MegaLD disks, and the TBC system isn't perfect either. I've had a shot at writing my own software de-muxing of Laserdisc RF, and my own software decoding of NTSC video. Both of them work to some degree, but both of them have their own different problems to the ld-decode software currently. A few months ago Chad Page was on a bit of a roll with ld-decode, and I was hitting a bit of a wall, so I shelved my efforts for awhile to leave him to run with it, and see where things ended up. I'm re-evaluating that now, seeing where he's up to, and looking at the problems I was having again to see if I can solve them.

In a nutshell, things are close, but not done yet. The amount of effort that has gone into this is truly insane. If I ever thought I'd have to learn how to demux an RF signal and decode NTSC video in order to achieve the outcomes of this project, I never would have even started it. As I post this though, I've got a ripping process running in my garage to re-dump the disks using the latest techniques and hardware. I'm still holding out hope to get some good decodes of the video happening in the next 6-12 months. If that doesn't occur, I'll most likely shove the terabytes (literally) of raw RF dumps on some file store just to get something out there in the interim. Those raw RF files will need to be preserved regardless, as the software decoding space will continue to evolve and you'll want to be able to re-run it from the source, but for actual emulation, the goal is a full digital data stream in a simple form (which I have), alongside a full analog video/audio stream in a simple video container (which I currently have as RF, but can't reliably decode to actual video yet).

[Nemesis]

If anyone's interested in reading a bit about the evolution of the software video decoding efforts, the best record is captured here:
https://forum.lddb.com/viewtopic.php?f=32&t=2671
This doesn't contain some of the more recent discussions and results, but it has a lot of the steps leading up to where we are now. I start joining in on the discussion in June 2017, and you can see a handful of WIP rips from MegaLD video there, as well as some other sources.

[Chilly Willy]

Nice summary. That's a LOT of work! ld-decode looks like a good long-term solution, but have you thought about a short-term makeshift of capturing the vblank data separate from the video? Pair that with a standard capture card might get you going until ld-decode works out the bugs.

[Nemesis]

I did think about that, but in itself that was challenging, as I'd need to construct a delay line to offset the vertical sync position, without throwing the horizontal sync out of alignment. I never attempted to build one, but I actively looked at how I might construct this kind of setup. It's clearly possible, and my Sony PVM monitors have an option built-in that allows me to delay the H/V sync pulses by a configurable amount, using which I can see the codes from the vblank region. I actually considered seeing if I could hook into internal signals in my PVM to make use of their delay feature. Another option was attempting to reconstruct the sync manually with the delay added using a micro-controller or FPGA. I didn't go too far down this route however before the Domesday Duplicator appeared, and I shelved traditional video capture attempts (which I'd spent quite a bit of time and money exploring) in favour of exploring the software decoding route. I'm not opposed to trying that kind of method again, but it feels like the software decoding route is really close at this point, and it has significant advantages.

[Nemesis]

Oh, on re-reading, if you just meant capturing the codes in the vblank region as some kind of data stream rather than image data, well that's technically possible now. Even though decoding the NTSC video from the RF signal is problematic still, the control codes are perfectly contained in the composite signal, and you could theoretically process that raw signal to extract them. I'm aiming to construct a full 525-line NTSC video frame from the interlaced signal though, like this:

You could start with an analog video rip from a traditional capture card, then combine that with reconstructed control code information from the extracted composite signal from the raw RF, to fake this kind of image by leaving the other border regions we don't generally care about blank. I'm more inclined to push on at this point and get the video decoding itself working. As you can see from this image (this one from my own software decoding efforts, not ld-decode), things are quite close.

[Sik]

Wait, control codes in vblank? What is being stored there? Or is this just standard laserdisc stuff?

For what matters, something that can dump all of vblank is going to be needed anyway for laserdiscs in general (not just Mega LD or LD-ROM²), since it turns out a good bunch of laserdiscs used teletext streams as a way to provide subtitles in multiple languages on a single disc.

[Chilly Willy]

Another good question is - is this standard vblank data as specified for closed caption and other uses?

https://en.wikipedia.org/wiki/EIA-608

[Nemesis]

I haven't bothered to compare the format and breadth of codes between the various standards, but there are features that are unique to Laserdiscs. Picture stop codes are an example, which cause the player to automatically pause when they decode a frame with a stop code. It's worth noting however, that vendors could easily have custom codes they've given meaning on particular platforms. The MegaLD disks for example can only be played on a single player produced by a single vendor. Pioneer could use whatever custom encodings they wanted in this context to achieve the goals of the platform. There are also laserdisc based arcade systems, which similarly rely control codes in the VBI region, and I personally don't have a clue what features they rely on and what format they use. That information is vital to emulating them though, and MAME had to deal with this same VBI issue when generating laserdisc rips long ago. Aaron Giles wrote a bit about this (https://aarongiles.com/old/?p=234), and he later shared his solution to ripping VBI data (https://github.com/aaronsgiles/VBIUtils). I attempted his method, but ran into a variety of issues, and ultimately I was worried about the completeness of the solution, as only a restricted range of data could be extracted from the VBI region, and even then only in a processed, filtered form that was provided by the capture hardware. I felt it was a good solution for the time in which he came up with it, but ultimately didn't go far enough to be considered complete for preservation purposes, and that we needed to do a bit better here. The MAME team is watching this space closely, as they'll be re-ripping all the current Laserdisc based arcade games once a proven solution exists.

[Nemesis]

Here's a picture from the dungeon showing a digital data rip in progress:

It takes about 24 hours to rip each side using this technique, as the Mega Drive isn't capable of pushing the sector data out as fast as it comes down from the disk, so I have to read the disk in "stripes" and recombine afterwards. I also aim for around a dozen passes of the subcode data so that I have enough to reliably error correct. Basically I leave the rig running all day, and each evening switch over to a new side or disk. Ripping the raw RF data is faster, but less automated. I'll probably make a modified version of the Domesday Duplicator software to directly interface with my MegaLD dumping program soon to assist in automation, but I'd still need a lot of passes to extract clean subcode data even with the RF decoding route, which I've already solved in my setup, so I'll probably stick to extracting the digital data through my original method, and leave just the analog video and audio to the software decoding route. Being able to "combine" multiple RF rips to get a cleaner, more accurate signal is an interesting idea that should be possible, and could render the digital ripping system I've rigged up redundant, but nobody's even attempted that yet.

[HardWareMan]

It looks like teletext scanlines.