Sorry I'm late to answer your question. I'm working full time in a demo for my Genesis game and I've so little time to talk.
This more or less is how the editor works:
- I import the frames and the editor chechs if there are duplicated frames.
- Then I go frame by frame setting up areas to cut. Each area is a console's sprite with the 4x4 patterns limitation. After I've set up all areas in each frame the editor can do its magic.
When I export an animation the editor do this:
- Cut every area in each frame and check if some areas are the same from a frame to the next one. If that happen the editor ignores that area when is going to load the next frame. This reduce the amount of patterns to load in each frame. If an area it's the same in the next frame but is in a different position the editor only save the new position.
- Alfter every area was cut the editor extract all patterns. To reduce the amount of patterns as low as possible the editor saves 2 kinds of load instructions: 1) load consecutive patterns or 2) jump from address to address to load a pattern that is the same as a previous one.
- In the end the editor generate instructions for setting up positions and sizes for each sprite in a frame.