Vectors, matrices and embeddings

Vectors

A vector is an ordered list of numbers, $\mathbf{v} = (v_1, v_2, \dots, v_n)$. In two or three dimensions you can draw it as an arrow from the origin; in machine learning it typically has hundreds or thousands of entries, and the arrow picture still guides intuition. Vectors add entry by entry, and multiplying by a number scales every entry:

The length (Euclidean norm) is $\lVert \mathbf{v} \rVert = \sqrt{v_1^2 + \dots + v_n^2}$, Pythagoras in $n$ dimensions.

The dot product

The dot product multiplies matching entries and adds them up:

It has a geometric meaning that makes it the workhorse of machine learning:

where $\theta$ is the angle between the vectors. Large and positive means they point the same way; zero means they are perpendicular; negative means they point apart. A single neuron computes a dot product between its weights and its input, and attention scores are dot products between queries and keys.

Matrices are transformations

A matrix multiplied by a vector gives a new vector. For a $2 \times 2$ matrix,

The best way to understand this is to ask where the basis vectors go. $(1, 0)$ lands on $(a, c)$, the first column, and $(0, 1)$ lands on $(b, d)$, the second column. Every other vector $(x, y) = x(1,0) + y(0,1)$ lands on the same combination of the columns. So a matrix is completely described by where it sends the basis vectors, and the grid of the plane is carried along with them: lines stay straight and parallel, and the origin stays fixed. That is what “linear” means.

Some things to try: a rotation has determinant 1 (areas are preserved); a reflection has a negative determinant (orientation flips, so the grid is mirrored); and “Project onto x” has determinant 0, squashing the whole plane onto a line. A zero determinant means information is lost, so the transformation has no inverse. Applying one matrix after another corresponds to multiplying them, $B(A\mathbf{v}) = (BA)\mathbf{v}$, which is why a stack of linear layers with no nonlinearity in between is no more powerful than a single layer.

Embeddings

An embedding maps each item, such as a word, a product or a user, to a vector, chosen so that geometry reflects meaning: similar items get nearby vectors. The vectors are not designed by hand. They are parameters learned by gradient descent, typically while training a model to predict context, such as which words appear near which. Word2vec (2013) popularised the idea with word vectors of a few hundred dimensions; modern language models learn embeddings of every token as their first layer.

Cosine similarity

To compare embeddings we usually ignore length and look only at direction:

It ranges from $-1$ (opposite) through 0 (unrelated) to 1 (same direction). Length is set aside because in many embedding models it tracks things like how frequent a word is, rather than what it means.

In the toy space, cat and kitten have cosine similarity 0.946, while cat and truck have −0.910. The “vector arithmetic” box shows the famous analogy trick: kitten − cat + dog lands exactly on puppy here, because the toy vectors were built so that “young version of” is the same displacement for both animals. In real embeddings such analogies hold only approximately, and are known to reflect biases in the training data.

References

1. T. Mikolov, K. Chen, G. Corrado and J. Dean, “Efficient estimation of word representations in vector space”, 2013. arXiv:1301.3781
2. G. Strang, Introduction to Linear Algebra, 6th ed., Wellesley-Cambridge Press, 2023.
3. 3Blue1Brown, Essence of linear algebra, which develops the “matrices are transformations” view visually.