An Introduction to Tensor Calculus

I saw a car today with the number plate TEN 50R, and before I'd registered the make of the car I had already read it. I do this with every plate I pass, and have done for as long as I can remember: the letters and digits arrive and my head starts rearranging them into a word, a date, a name, or a joke at somebody's expense (usually the driver's). This one didn't need much rearranging. The 5 becomes an S, the 0 becomes an O, and TEN 50R becomes tensor.

Somebody else would have read it differently. The owner presumably paid good money for it to mean something, and I'd bet that something wasn't tensor calculus. It could be initials and a birthday, a nickname, or nothing at all, just the cheapest plate left on the website that day. The reading was mine. It depended entirely on who happened to be standing on the pavement, and a different person on the same pavement would have walked away with a different word.

That dependence is a good place to start, because the objects this post is about are defined by the opposite property. A tensor is a thing whose meaning survives a change of viewpoint. Two observers can describe one with completely different lists of numbers and still be talking about the same object, and the rules for translating one list into the other are, more or less, the whole subject.

What survives when you change how you look at something? Which numbers are properties of the world, and which are properties of the person measuring it? If I rotate my axes, which of my measurements change and which stay put? And if the measurements all change, what is it that stays the same underneath them?

There's a practical reason to care as well. The word has become common currency in tech, mostly through machine learning, where the most popular libraries are built around it (one of them, TensorFlow, is literally named after it). The trouble is that the word is used in two different ways. In machine learning, a tensor is a multidimensional array of numbers: a list, a grid, a stack of grids, and so on upwards. In physics and mathematics, a tensor is an object defined by how its components change when you change coordinates, and the array is only what it looks like in one particular coordinate system. Every tensor can be written as an array. Plenty of arrays are not tensors.

If you're heading into AI or ML, you will spend your days manipulating tensors in the first sense, and the index notation and the rules for combining them carry straight over from the second. The deeper machinery (covariant derivatives, curvature) turns up far less often in day-to-day ML work; it lives mainly in general relativity, continuum mechanics and differential geometry. I still think it's worth learning properly, because it trains a habit that pays off everywhere: keeping the thing and its description apart.

So this post builds up from single numbers to the physicist's tensor, goes far enough into tensor calculus to reach the Christoffel symbols and the geodesic equation, and then comes back to machine learning to show which parts of that machinery actually turn up there, with a short code example.

#Scalars

Start with a single number. Linear algebra will tell you a number can be thought of as a matrix, and as a picture of how it's stored that's fine, but it misses what makes a scalar a scalar in the tensor sense. A scalar is a quantity every observer agrees on, whatever coordinates they choose to describe it with.

Temperature at a point in a room is a scalar. Set up your axes pointing at the door, I'll set mine pointing at the window, and when we both read the same thermometer we get the same answer. The distance between two points is a scalar. So is the rest mass of a ball.

Now take the velocity of that ball as it rolls across the floor and ask for its -component. That is also a single number, and it is not a scalar. Your -axis points at the door, mine at the window, and we'll write down different numbers for the same ball at the same instant. Nothing about the ball differs between us; only the axes do. So "a single number" and "a scalar" are different things, and the difference is whether the number belongs to the object or to the observer's choice of description.

Formally, if we relabel every point by switching from coordinates to new coordinates , a scalar field obeys:

The value at a point is unchanged; only the label on the point has moved. This is a tensor of rank , an object with no indices at all, and the simplest example of a reading everyone agrees on.

#Vectors

Vectors usually arrive as a list of numbers, or , written as a row or a column, and the first thing worth unlearning is that the list is the vector. The vector is the arrow. The list is what the arrow looks like once you've chosen a set of axes and asked how far it reaches along each of them.

Velocity is the standard example, a magnitude and a direction held in one object, and it's a vector whether or not anything with mass is attached to it (multiply by the mass and you get momentum, , which is another vector). A ball rolling at 5 metres per second towards the window is one physical fact. In my axes its velocity might read . Rotate the axes and the same arrow reads , or , or . The components changed every time, the arrow never did, and its length, , came out the same in every case. That length is a scalar, which is exactly why everyone agrees on it.

So a vector is pinned down by how its components respond when the axes change, and once you start asking that question carefully, vectors split into two kinds.

Imagine changing the units along one axis from metres to centimetres. Each basis vector (the little arrow marking one unit along that axis) is now a hundred times shorter, so to describe the same displacement you need a hundred times as many of them, and the component grows by a factor of . The components move opposite to the basis, which is why these are called contravariant vectors. Displacement and velocity are the classic examples, and they're written with an upper index, . Under a general change of coordinates from to , they transform as:

That formula uses the Einstein summation convention, which I'll use throughout: an index that appears once up and once down in the same term is summed over all its values. Here runs over every coordinate, so each new component is a weighted mix of all the old ones.

Now take the gradient of the temperature in the room, how fast the temperature changes as you move along each axis. Its components are measured in degrees per metre. Switch that axis to centimetres and the same physical gradient becomes a hundred times smaller per unit, so the component shrinks along with the basis. These are covariant vectors (also called covectors or one-forms), written with a lower index, , and they transform with the inverse of the factor above:

The reason for keeping the two apart shows up as soon as you combine them. Walk through the room with velocity and ask how quickly the temperature you feel is changing. The answer is the velocity contracted with the gradient:

One transformation factor and its inverse cancel, and what's left is a scalar. That has to be the case, because a thermometer in your pocket doesn't care which way anyone drew their axes.

If none of this came up at school, there's a reason. With flat space and perpendicular axes of unit length (ordinary Cartesian coordinates) and only rotations allowed between them, the two transformation rules produce identical numbers, so the distinction vanishes. Writing one kind as a column and the other as a row is a handy way to keep them apart in matrix notation, but the difference itself lives in the transformation rule, and the object that converts one kind into the other is the metric, which we'll meet shortly.

#Matrices

A matrix is a rectangular grid of numbers, rows by columns (square only some of the time), and it's usually introduced as a machine that acts on vectors: feed a vector in, get another vector out. One of the first you'll meet is the rotation matrix, which turns any vector in the plane anticlockwise by an angle :

Feed it the vector and out comes , the same arrow swung round by . Every linear transformation (rotations, stretches, shears, reflections, projections) can be written as a matrix in the same way, once you've picked a basis.

That clause again: once you've picked a basis. A matrix is to a linear map what a list of components is to an arrow. It's the map described in one particular set of axes, and if you change the axes the grid of numbers changes with them.

Here's where it gets interesting. The same grid of numbers can describe two completely different kinds of object. Take the identity grid, ones on the diagonal and zeros elsewhere. Read one way, it's the linear map that does nothing, handing back whatever vector you feed it. That object has one upper and one lower index, . Read another way, it's the metric of the flat plane in Cartesian coordinates, the object that takes two vectors and returns their dot product, and therefore every length and angle on the plane. That object has two lower indices, .

Switch to polar coordinates and the two part company. The do-nothing map is still the identity grid in any basis whatsoever. The metric, though, becomes:

The is there because a small step in angle covers more ground the further you are from the centre. Same plane, same object, different grid. Hand someone the identity grid on its own and they cannot tell you which of the two they're holding. Only the transformation law can settle it.

#Tensors

Which brings us to the general definition. A tensor of type is an object with upper indices and lower ones, whose components pick up one factor of for every upper index and one factor of the inverse, , for every lower index when you change coordinates:

It looks heavier than it is. Every index simply carries its own copy of the vector rule from earlier, upstairs indices the contravariant one and downstairs indices the covariant one. Everything we've met so far slots in: a scalar is type , a velocity , a gradient , a linear map , a metric . The total number of indices, , is the tensor's rank.

The picture of a tensor as a matrix stacked into further dimensions, , is a fair picture of the storage. In dimensions a rank- tensor has components, so in four-dimensional spacetime a rank- tensor has and a rank- tensor has . The dimensions also needn't all match, at least for arrays in general. What makes the stack a tensor, though, is the rule above, and the rule is where all the physics comes from.

Its most important consequence is easy to state. If a tensor equation holds in one coordinate system, it holds in all of them, because both sides transform in exactly the same way. In particular, a tensor whose components are all zero in one frame is zero in every frame. This is why physicists insist that laws of nature be written as tensor equations, a requirement known as the principle of general covariance: a law that held for me and failed for you, merely because we drew our axes differently, wouldn't be telling us anything about the world.

The metric deserves a second look here, because it does more than measure distances. It's the bridge between the two kinds of index. Contracting with the metric lowers an index, and contracting with its inverse raises one:

In flat Cartesian coordinates the metric is the identity grid, so raising and lowering change none of the numbers. That's the deeper reason the two kinds of vector look the same at school: the metric doing the converting was the identity all along.

A few tensors worth knowing by name:

• The stress tensor in a solid, rank . To describe a force acting inside a material you need two directions: the way the surface you're cutting through faces, and the way the force on that surface points.
• The electromagnetic field tensor, , rank and antisymmetric, which bundles the electric and magnetic fields into a single object. What one observer measures as a purely electric field, a moving observer measures as partly magnetic, and the tensor is the thing both of them are describing.
• The Riemann curvature tensor, rank , which measures how curved a space is. We'll get to it once we've built the tools to define it.

#Where the Calculus Starts

Everything so far has been tensor algebra: objects sitting at a single point and the rules for combining them there. The calculus begins when tensors vary from place to place, as fields spread across a space, and you want to know how fast they change. That turns out to be harder than it sounds.

Differentiating a scalar field is painless. Its rate of change along each coordinate, , transforms exactly like a covariant vector, as the temperature gradient did. Try the same thing with a vector field, though, and the result refuses to behave:

The first line is the tensor rule. The second is an intruder, built from the second derivatives of the coordinate change itself, and its presence means the plain derivative of a vector is not a tensor. Measure it in one coordinate system, translate the answer into another, and you get something different from what you'd have measured there directly.

The reason is worth slowing down for. A derivative compares the field at one point with the field at a neighbouring point. For a vector, that means comparing components, and the components at each point are measured against the basis at that point. In Cartesian coordinates the basis is identical everywhere, so the comparison is fair. In polar coordinates it isn't: the arrow pointing "outwards" at the top of a circle points a different way from the one at its side, and the "around" direction stretches the further out you go. Hold a single arrow pointing due east and walk around a circle with it, and its polar components change continuously even though the arrow never does. A plain derivative can't distinguish "the vector changed" from "the axes changed underneath it".

The fix is to add a correction that accounts for how the basis itself moves, giving the covariant derivative:

The coefficients form what's called a connection, since they connect the basis at one point to the basis at the next. Here's the curious part: is not a tensor either. It transforms with an intruder term of its own, and that term is precisely the one needed to cancel the intruder in the plain derivative. Two objects that each fail the test add up to one that passes.

There are many possible connections, but geometry hands us a natural one. Demand that carrying vectors around with the connection never changes their lengths or the angles between them (the covariant derivative of the metric vanishes, ), and that it introduces no twisting of its own (), and is fixed completely by the metric. These are the Christoffel symbols, and the connection they define is the Levi-Civita connection:

It's worth computing one, and the flat plane in polar coordinates is the cheapest example there is. With and , only two independent symbols survive:

Both have plain meanings. The first says that as you move around the circle, the "around" basis arrow turns inwards, towards the centre, at a rate proportional to . The second says that as you move outwards, that same arrow grows in proportion to how far out you are. And notice the plane here is perfectly flat. Nonzero Christoffel symbols tell you your coordinates are curved, which isn't the same thing as the space being curved.

With the covariant derivative in hand, we can say what a straight line is in any coordinates, and later in any space at all: a curve whose direction never changes as you travel along it, meaning its own tangent vector is carried along without rotating. Write that condition out and you get the geodesic equation:

In Cartesian coordinates on a flat plane every is zero, what's left says the acceleration is zero, , and you have the straight line from school. In general relativity this equation describes how anything moving under gravity alone travels through curved spacetime. The planets' orbits are geodesics, and the whole of what we used to call the gravitational force has moved into the term.

That leaves the question of how to tell whether a space is genuinely curved or just described in awkward coordinates. At any single point you can always choose coordinates that make every vanish there (physically, this is a freely falling frame, in which gravity locally disappears). What you can't remove everywhere, if the space is truly curved, is the way changes from point to point. The Riemann curvature tensor collects exactly that leftover:

Being a tensor, it's either zero in every coordinate system or zero in none. Grind it out for the polar plane and every component vanishes, as it must for a flat surface wearing curved coordinates. Do the same on the surface of a sphere and it survives whatever coordinates you choose, because no coordinate system can flatten a sphere (ask anyone who's tried to make a map of the Earth).

General relativity is built from contractions of this object. Contract Riemann once and you get the Ricci tensor, , contract again and you get the Ricci scalar, , and Einstein's field equations set a combination of the two equal to the matter and energy present:

It's a tensor equation, so it holds for every observer in every coordinate system, which is precisely the reading everyone agrees on that this whole subject was built to find.

#What Machine Learning Calls a Tensor

Open PyTorch and a tensor is an -dimensional array of numbers, plus some bookkeeping: what type the numbers are, which device holds them, and a record of the operations that produced them so the library can differentiate through them later. A batch of 32 colour images, each pixels, is a tensor of shape : image, colour channel, row, column. Four indices, so rank by the counting definition.

Nobody, though, is going to transform the batch index under a change of coordinates. Its four axes mean entirely different things and have different sizes, and there's no transformation law tying them together. By the physicist's definition, this is an array that happens to be called a tensor. That's fine (names are cheap, and the computer certainly doesn't mind), as long as you know which meaning you're dealing with.

What carries over intact is the notation, and with it a great deal of the thinking. The Einstein summation convention, where a repeated index means "multiply and sum", is implemented almost word for word as einsum:

import torch

# A batch of 32 sequences, 128 tokens each, 512 features per token
x = torch.randn(32, 128, 512)

# A linear layer's weights: 512 features in, 256 out
W = torch.randn(256, 512)

# y[b, t, o] = sum over i of W[o, i] * x[b, t, i]
# the repeated index i is summed away, exactly as in Einstein notation
y = torch.einsum('oi,bti->bto', W, x)
print(y.shape)  # torch.Size([32, 128, 256])

# The core of attention: score every token against every other token
q = torch.randn(32, 128, 64)
k = torch.randn(32, 128, 64)
scores = torch.einsum('bqd,bkd->bqk', q, k) / 64 ** 0.5
print(scores.shape)  # torch.Size([32, 128, 128])

# Contracting an index with itself: the trace, T^mu_mu
T = torch.randn(4, 4)
print(torch.einsum('ii->', T), torch.trace(T))  # the same number twice

Read the strings as index notation and they're nothing new. The first is

where appears on both inputs and not in the output, so it's summed, which is a matrix multiplication applied to every token in every sequence at once. The second is

where the feature index is summed away to leave a grid of similarities per sequence (divided by to keep the numbers in a sensible range), which is the heart of the attention mechanism in a transformer. The third contracts a tensor's two indices with each other to give a single number, . Training these models leans on the same habit: backpropagation is the chain rule applied layer by layer, which in index form is a long chain of contractions of derivatives, and the libraries are built to carry it out efficiently.

The full physicist's machinery does turn up in a few corners. The best known is information geometry, which treats the space of a model's possible parameters as a curved space with its own metric, the Fisher information, . In that light, ordinary gradient descent has a quiet flaw. The gradient of the loss, , is a covariant vector (it has a lower index, like the temperature gradient), while a step in parameter space, , is a contravariant one, and plain gradient descent adds one to the other as if the metric were the identity, meaning it assumes the parameter space is flat and Cartesian. The natural gradient, introduced by Shun-ichi Amari in 1998, raises the index properly with the inverse Fisher metric, , before taking the step:

The same idea, in approximate and cheaper forms, sits behind several modern optimisers. And a growing field, geometric deep learning, builds networks whose outputs transform correctly when their inputs are rotated or shifted, which is the transformation law put to work. So if you're going into AI, you'll handle the array kind of tensor every day, the notation will be second nature within weeks, and the rest of the subject will be waiting for you in the places where the field is still being worked out.

#Back on the Pavement

The plate said tensor to me and almost certainly something else to the person who bought it, and neither of us was wrong. A reading of a plate is a set of components: real, useful, and entirely dependent on where the reader is standing. What the subject asks of you is to hold the description and the thing described apart, and to know which of your numbers belong to the world and which belong to your own choice of axes.

Physics spent the better part of three centuries learning to write its laws in a form every observer would sign off on, and general relativity is where that effort reached its conclusion. I suspect the habit is worth more outside physics than in it. Most of the arguments I've watched go nowhere (a good number of them, admittedly, with myself) were two people quoting different components of the same thing at each other, each convinced the other was misreading it. Before defending a number, it's worth asking whether you'd still get it if you stood somewhere else.

I'll keep reading every plate I pass. I'll just be a little clearer, from now on, about whose reading it is.

If you'd like to go further, I've uploaded my own revision notes from the Space, Time and Gravity module I took at university in 2014, along with the one-page formula sheet I kept beside them: STG revision notes and STG formula sheet. They run from the equivalence principle through tensor calculus and the Schwarzschild metric to cosmology, and anyone working through a general relativity module may find them useful.