I've read about people with the exact same issue you describe. Keys work in other use cases (like, software that use the Amiga keys for keyboard commands) etc. and I'm willing to bet your keys all work fine independently.
If that's the case, the culprit was sometimes the Amiga's keyboard microcontroller, or mpu.
Some related data sheets: