Active Disassembly - Post Processor

Official support forum for the Exodus Emulation Platform

Moderator: Nemesis

Post Reply
ryanfaescotland
Very interested
Posts: 53
Joined: Mon Feb 09, 2015 10:46 pm
Contact:

Active Disassembly - Post Processor

Post by ryanfaescotland » Sun Mar 31, 2019 9:33 pm

I'm still playing around with the active disassembler for Megadrive games and have just created my second disassembly using it.

I was thinking there is a few things I'd like to see it doing to the file it generates, some of which are mentioned in the feature requests as well, and I was thinking about how they would best fit in with the Exodus project as a whole.

Best I could really come up with is Post Processing, mostly because I'm a wimp and wouldn't want to integrate anything directly in for fear of making a huge mess of the code base or completely perverting the original spirit of the project!!

Here is a few of the features I'd like to see, and I'd be interested to know if you, Nemesis, or any of the other Exodus users reckon they would fit in with the existing tool or be handy as a 'post active disassembly' step that can be ran on the file Exodus generates:
  • Generate meaningful header labels,
  • Substitute labels through from an external mapping file, plus those generated in the header,
  • Substitute known addresses for constants (VDP, I/O, Z80 and so on),
  • Split data and code into separate files
Anyone any other suggestions or thoughts on this?

Nemesis
Very interested
Posts: 791
Joined: Wed Nov 07, 2007 1:09 am
Location: Sydney, Australia

Re: Active Disassembly - Post Processor

Post by Nemesis » Tue Apr 02, 2019 4:48 am

Thanks for your suggestions and feedback. I'm glad you're finding the current features useful. There is certainly a lot that could be done to the output from active disassembly to make the generated code more useful. One of the challenges though is keeping things generic and platform independent. I've got a lot of plans around arcade hardware I'd like to support for example, a fair few of which also use an M68000 processor. The current implementation of the M68000 core is platform independent, and so is the active disassembly feature and subsequent analysis. This means those features will work correctly without any changes on, for example, the three M68000 processors on a Sega Y-board running G-LOC, just as well as they will in the Mega Drive. What this means is that nothing in the base analysis process can depend on system specific knowledge. I do plan to add support to "trace through" banking hardware and the like so that bankswitching doesn't interfere with active disassembly, but nuances like recognizing system-specific register banks and binary headers would need to stay out of the cores themselves.

As you suggest, a post-processing step is a potential solution to the obvious desire to do additional system-specific cleanup to a generated disassembly, without baking that logic into areas of the code that should remain platform independent. To be honest, I always imagined a lot of that work being farmed out to other tools written by other people. There's currently an option to generate an IDC script for example, to use in IDA Pro. I imagined you might import the initial disassembly with that script, then run one or more system-specific cleanup scripts (IE, "CleanMegaDriveDisassembly.idc"), which performs the kind of tasks you've mentioned, like labelling the header fields, known hardware registers and address regions, and possibly scattering in comments or better naming elements of the disassembly. Even with the direct asm output though, nowadays I'd think some well written python scripts might be the solution. Do you think that kind of approach is viable here? Even if you needed to feed that kind of script the original ROM, plus the asm output, plus the "text output" that contains the raw analysis outcomes, that should be enough for an independent python script to implement the kind of features you've described. It's possible "standard" scripts could be added to the actual Exodus release and shipped with it, but generally I'd consider this add-on functionality that other people would create and distribute.

I guess that leads to the question, would you like to try your hand at cutting some code that could do this? It's possible I could add in some kind of "one-click" support for running user-defined external post-process steps on the disassembly output, but I'd like to see one or two of them appear publicly first, so I could get a feel for what input they'd need and how they'd operate in practice.

ryanfaescotland
Very interested
Posts: 53
Joined: Mon Feb 09, 2015 10:46 pm
Contact:

Re: Active Disassembly - Post Processor

Post by ryanfaescotland » Tue Apr 02, 2019 12:17 pm

You've hit the nail on the head about what I mean about 'perverting the original spirit of the project'. My concern would be implementing something that unknowingly creates something Megadrive specific or dependant, losing the genericness / independance I can see you are aiming for. This would be much less of a worry developing at 'arms length' though as an independent project that just works on the output from Exodus rather than being a direct part of it. So yeah, I'm definitely up for writing a standalone tool of some sort, and if it goes well it would be pretty neat to have it integrate as an automatic post processing step.

I need to disappear for Real Life stuff again but I'll hop back on for an hour or so tonight.

Perhaps you could fill in a couple of details for when I do though - Does the Active Disassembler target any specific assembler with it's output or is it just meant to be a generic 68k output? Assuming it is generic, what assembler do you use afterward to test the output and does it build immediately?

Ok, I gotta run. Will talk more later.

Oh, side note, I don't have IDA (I know of it and what it is though) so not sure what it does or how it does it other than the description you've just gave me.
Last edited by ryanfaescotland on Fri Apr 05, 2019 10:23 pm, edited 1 time in total.

Nemesis
Very interested
Posts: 791
Joined: Wed Nov 07, 2007 1:09 am
Location: Sydney, Australia

Re: Active Disassembly - Post Processor

Post by Nemesis » Wed Apr 03, 2019 1:23 am

The assembly output is meant to be generic, however I'm not sure there's such a thing as "standardised" 68000 assembly. I do all my testing and compilation with asm68k.exe, which is an oldschool assembler from long ago. IDA (https://www.hex-rays.com/products/ida) is an absolutely incredible disassembler, the professional version being the one you want to get you all the good features and a wide range of processor support. It is also very expensive, and has historically been somewhat difficult to purchase as an individual. If you don't already have experience using it, it's probably better to stick to raw asm, as active disassembly takes a lot of the hard work out of the disassembly process itself, and IDA has a bit of a steep learning curve.

ryanfaescotland
Very interested
Posts: 53
Joined: Mon Feb 09, 2015 10:46 pm
Contact:

Re: Active Disassembly - Post Processor

Post by ryanfaescotland » Fri Apr 05, 2019 11:14 pm

Here I am in 2016 complaining, on this very forum no less, about the price of IDA! :lol: viewtopic.php?f=7&t=2264&p=28549#p28545

I'm at a bit of a crossroads at the moment, not 100% sure what route I want to go down so apologies but this post is a little bit of a brain dump. I could do with a project I could use as a bit of a demonstration of my coding ability, meaning it would have to be C# and preferably something with multiple parts, typical frontend, backend sorta stuff, maybe even a little storage in there. At the same time however, I want to develop something useful and enjoyable. So, although a post processor may be doable in python or the like I'd still be looking to implement it in C#, even if it is OTT.

With the over-the-topness in mind I was trying to think of other tools that could be useful in post processing, but I keep feeling a bespoke / new development isn't the way to go:
  • An up-to-date 68k assembler might have been nice, but it looks like vasm has that covered.
  • Then I was thinking an IDE specifically for Megadrive development would be good. But the more I think about this the more I think it would end up as either A) an emulator or B) more suitable as a Visual Studio Code Extension like Amiga Assembly
So, basically what I'm getting at is that just now I just don't know what direction I want to go in. I want to do something but I'll need to mull it over for a little bit (just a little mind you lest I lose the drive to do something).

I'll chew on it for a bit and come back to it.

TheMVRules
Interested
Posts: 11
Joined: Wed Apr 07, 2010 6:00 pm
Location: Linköping, Sweden

Re: Active Disassembly - Post Processor

Post by TheMVRules » Sat Apr 06, 2019 11:37 am

You don't have to complain about IDA anymore, Ghidra is free, supports the 68k/z80, has a decompiler (that works for said processors) and has been used by NSA for years.

Regarding vasm, I'm not sure if it's a good choice for development as its license expressly forbid commercial use (and they say that applies to generated binaries too).

ryanfaescotland
Very interested
Posts: 53
Joined: Mon Feb 09, 2015 10:46 pm
Contact:

Re: Active Disassembly - Post Processor

Post by ryanfaescotland » Sat Apr 06, 2019 12:51 pm

Thanks TheMVRules, that's both interesting to know. Will keep the licensing of vasm in mind for sure.

I think, to start at least, I will go the VSCode extension route. The ideas I have in mind would be helpful to me at least and I think the wider community. Also, if developed with separation in mind I should be able to pull the bits that would be useful standalone into a separate form that could be integrated else where, i.e. a standalone postprocessor for Exodus generated disassemblies.

Chilly Willy
Very interested
Posts: 2984
Joined: Fri Aug 17, 2007 9:33 pm

Re: Active Disassembly - Post Processor

Post by Chilly Willy » Sat Apr 06, 2019 1:52 pm

I use the Amiga program "ReSource" to disassemble MD games. It doesn't handle Z80, but that's rarely an issue. It's VERY easy to deal with 68000 code and data mixed as it is in MD roms. Well, I always thought it was easier than IDA. And cheaper.

http://amiga-dev.wikidot.com/tool:resource

Nemesis
Very interested
Posts: 791
Joined: Wed Nov 07, 2007 1:09 am
Location: Sydney, Australia

Re: Active Disassembly - Post Processor

Post by Nemesis » Sat Apr 06, 2019 4:16 pm

TheMVRules wrote:
Sat Apr 06, 2019 11:37 am
You don't have to complain about IDA anymore, Ghidra is free, supports the 68k/z80, has a decompiler (that works for said processors) and has been used by NSA for years.
Oh snap! I hadn't heard about this, just had a 5 minute play around. Damn, that is a nice tool, and the decompiler they've got works so quickly, and I think does a better job than the one in IDA. It'd cost almost $5000 to buy a fixed single user license of IDA that's comparable to what the NSA have released here. IDA developers must be ticked off though, a nation state just stepped all over their business model. It was never commercially viable for a competitor to come up in this market as it's too small a niche, and a community lead open-source tool would have taken an age to come close, but the NSA can afford to have a team of crack developers work on this full time forever and it won't even be a blip in their total budget, hell, it's probably counted as a budget saving measure to make their other efforts more productive. If Hex-Rays was US based, you'd probably see a lawsuit come out of this.

On the other hand, I'm actually more suspicious of this program than I am of the cracked version of IDA I torrented. Does anyone else feel a bit weird about willingly installing a large complex piece of software supplied directly by the NSA? If nothing else, I'd expect my IP address just went into a database when I clicked their download link! :)

Chilly Willy
Very interested
Posts: 2984
Joined: Fri Aug 17, 2007 9:33 pm

Re: Active Disassembly - Post Processor

Post by Chilly Willy » Sat Apr 06, 2019 4:48 pm

TheMVRules wrote:
Sat Apr 06, 2019 11:37 am
You don't have to complain about IDA anymore, Ghidra is free, supports the 68k/z80, has a decompiler (that works for said processors) and has been used by NSA for years.
Completely missed this. Thanks for pointing it out - I'll have to try it.
Regarding vasm, I'm not sure if it's a good choice for development as its license expressly forbid commercial use (and they say that applies to generated binaries too).
You should talk to Volker about that. I was able to talk him into allowing the use of vbcc/vasm for even commercial use on Jaguar binaries (not the compiler, just the binaries generated by the compiler/assembler/linker).

Nemesis
Very interested
Posts: 791
Joined: Wed Nov 07, 2007 1:09 am
Location: Sydney, Australia

Re: Active Disassembly - Post Processor

Post by Nemesis » Mon Apr 08, 2019 11:36 pm

I've had a solid play around with Ghidra, and written an experimental active disassembly script export function for it. There are a few dealbreakers with using it for ROM disassembly currently though:

1. There appears to be no way to cleanly export its disassembly in a compilable form. The tool was designed around analysis of compiled software, but actually exporting generated assembly code so you could recompile from it doesn't seem to have been an important consideration with it as it's been developed to date. The 68000 disassembly doesn't seem to even represent "pointers" and data blocks in a form that 68000 assemblers I've seen would recognise.

2. It doesn't seem to support relative offsets currently. This means you can't seem to define, for example, a 16-bit integer value that's used as a relative offset to another location, with a given base address. This is essential in order to be able to represent offsets in the code in a form that makes it recompilable after changes.

It is of course open source, so someone might come along and address these issues in the future, but currently it doesn't look like a viable platform to use for ROM disassembly.

Chilly Willy
Very interested
Posts: 2984
Joined: Fri Aug 17, 2007 9:33 pm

Re: Active Disassembly - Post Processor

Post by Chilly Willy » Tue Apr 09, 2019 3:58 pm

Sounds like I should stick with ReSource for MD disassembly. :D

It handles all that rather well. You can't disassemble Z80 in it, but it does 680x0 like a charm. You'd have to make the Z80 parts binary blobs.

tryphon
Very interested
Posts: 316
Joined: Sat Aug 17, 2013 9:38 pm
Location: France

Re: Active Disassembly - Post Processor

Post by tryphon » Tue Apr 09, 2019 7:48 pm

Do you have any tutorial about ressource ?

Chilly Willy
Very interested
Posts: 2984
Joined: Fri Aug 17, 2007 9:33 pm

Re: Active Disassembly - Post Processor

Post by Chilly Willy » Tue Apr 09, 2019 11:08 pm

The manual is online here: http://www.pusher.dk/amiga/resource/manual/

It's not exactly a tutorial, but it does show you how to get started, and isn't hard to follow.

There's also a python 680x0 disassembler someone is making to replace ReSource found here: https://github.com/rmtew/peasauce

Haven't tried it, so I'm not sure how good it is. I've got ReSource in an E-UAE work folder that I've used for years. When I made patch files for games for NeoMyth, I used ReSource to find what needed changing.

tryphon
Very interested
Posts: 316
Joined: Sat Aug 17, 2013 9:38 pm
Location: France

Re: Active Disassembly - Post Processor

Post by tryphon » Wed Apr 10, 2019 6:36 am

Thanks.

I downloaded Peasauce last week, trying to play with its source. Didn't know it was aimed to replace resource though.

Post Reply