Matrix

matrix multiplication wasn't invented for neural networks. Neural networks inherited it from a mathematical idea that is centuries old.

Matrices represent transformations. Matrix multiplication represents applying/composing transformations.