The X/Y overflow flags are supposed to be set if the mouse is moved over a distance greater than can be measured. I can't get
them to become set, maybe the mouse supports a fairly wide range of movement.
Stupid question, what's the range actually reported by delta values in the mouse? We were doing some tests with PC mouses, and it turns out they like to clamp the range to -127 to +127 (effectively rendering the overflow flags unused, as the delta never manages to become large enough). It would be nice to know what the Sega Mouse and the Mega Mouse do, maybe their behavior happens to be similar.
Also I imagine a ball-based mouse is going to have a harder time to register a sudden motion like these tests would be doing (due to inertia).
If overflow is ever set, you ignore x or y (depending on which overflowed), and use +/-256. The sign bit is separate from x and y, so they're actually 9-bit ranges, so -256 to +255.