What is still missing

The theories in this course account for about 5% of the universe, cannot be combined with each other, and get one prediction wrong by a factor of 10¹²³. That is the honest state of physics, and it is a better place to end than a tidy one.

now20 min

By the end you should be able to

List the major open problems, and distinguish which kind of problem each one is
Say why quantum gravity has not been tested rather than merely not solved
Say what the vacuum energy discrepancy is, and why it is the worst prediction in science

This course began in 1831, with a bookbinder’s apprentice holding a coil of wire and watching a needle twitch. It ends with seventeen fields and a theory that predicts the magnetic moment of the electron to ten significant figures. Every step in between was forced by a measurement. Nobody in this story was asked to take anything on trust — not the reader, and not the physicists, several of whom spent years refusing to believe results that turned out to be correct.

The CMS detector at CERN, a vast cylindrical instrument in red and silver, with people at its base for scale. — CMS, at the Large Hadron Collider. An instrument this size exists because the Standard Model is the most precisely tested theory ever written — and because everyone involved knows it cannot be the last word.

One: measured, and unexplained

The experiment

Galaxies that rotate too fast

Vera Rubin and Kent Ford · 1970 – 1980 · Carnegie Institution; Kitt Peak and Lowell Observatories

The question: How fast do stars orbit at the outer edges of spiral galaxies, compared with the speed the visible matter would require?
The apparatus: A spectrograph sensitive enough to measure Doppler shifts of starlight at the faint outer edges of galaxies, beginning with Andromeda. Ford’s image-tube spectrograph made it possible to work far out into the disc, where nobody had been able to measure before.

Theory predicted

Keplerian falloff. Almost all the light is near the centre, so if light traces mass, outer stars should orbit more slowly — the way Neptune moves more slowly than Mercury.

They measured

Flat rotation curves. Outer stars move about as fast as inner ones, in galaxy after galaxy. The implied mass keeps increasing with radius long after the light has run out.

How sure could they be? The conclusion no longer rests on rotation curves alone: gravitational lensing, the Bullet Cluster — where the mass is measurably in a different place from the visible gas — and the peak structure of the cosmic microwave background all give a consistent dark matter density.

Why it mattered

There is roughly 5.5 times more matter than there is visible matter, distributed in a halo rather than a disc, and it is not made of anything in the previous lesson. Fritz Zwicky had inferred something similar from the Coma cluster in 1933 and been largely ignored for forty years.

You might think

Dark matter is probably just ordinary matter that is too dim to see — cold gas, dead stars, black holes.

Actually

This was taken seriously and has been ruled out, which is a better outcome than it sounds. Big Bang nucleosynthesis fixes how much ordinary matter exists, because the amount of helium, deuterium and lithium produced in the first minutes depends on it — and the answer matches the visible matter, not the total. The cosmic microwave background gives an independent measurement of the same quantity from the relative heights of its peaks, and agrees. So dark matter is not dim ordinary matter. Searches for dim ordinary matter in the form of compact objects have also come up short. Whatever it is, it is not made of the fields in the previous lesson.

Neutrinos have mass. The Standard Model, as originally written, said they had none. We know because they change type in flight — a muon neutrino arriving as an electron neutrino — and that oscillation is only possible if the different types have different masses. Super-Kamiokande showed it for atmospheric neutrinos in 1998; SNO settled the solar neutrino problem the same way in 2001. But oscillation measures mass differences. So: at least one neutrino weighs 0.0495 eV or more, and the direct KATRIN experiment says none weighs more than 0.8 eV. Between those the answer is unknown. For scale, the electron is at least ten million times heavier than the heaviest neutrino, and nobody knows why.

There is more matter than antimatter, and there should not be. In the early universe, for roughly every billion antiparticles there was about one extra particle. Everything else annihilated. That leftover excess — around one part in a billion — is every star, every planet, and everyone reading this. The Standard Model does contain a mechanism that treats matter and antimatter differently. It was found in kaons in 1964, it is real, it has been measured carefully — and it is far too small, by many orders of magnitude, to produce the excess we observe. We exist in greater quantity than the theory permits.

Two: two theories that disagree

General relativity and quantum field theory are both superbly confirmed, and they are not compatible. Relativity gave us Mercury’s perihelion to the arcsecond, the deflection of starlight, the GPS correction, and a gravitational wave detected across 1.3 billion light years. Quantum field theory gave us ten digits of the electron’s magnetism. Try to quantise gravity the way electromagnetism was quantised and the perturbation series produces infinities that cannot be absorbed into a finite number of measured parameters. The trick that made QED work does not work here.

Three: a number that is simply wrong

Back in the vacuum lesson, this was set aside for the end. Every mode of every field has a ground-state energy of ½ħω. Add them up out to the Planck length and empty space should carry an energy density of about 4.6 × 10¹¹³ J/m³. Empty space does carry an energy density. It is what makes the expansion of the universe accelerate, it was discovered in 1998, and it has been measured: about 5.4 × 10⁻¹⁰ J/m³.

Four: things that are merely unsatisfying

Two complaints about the Standard Model are of a different character from everything above. It has 26 free parameters. Masses, mixing angles, coupling strengths — measured, inserted by hand, unexplained. The Higgs mass is 17 orders of magnitude below the Planck scale, and quantum corrections would naturally drag it upwards, so something appears to be cancelling to many decimal places. This is the hierarchy problem. Neither is an observation that disagrees with theory. Both are the feeling that a good theory ought not to look like this.

Which is worth ending on, because this course has already shown what a null result can do. The single most consequential experiment in these forty-six lessons found nothing. Michelson and Morley looked for the ether and measured no effect at all, and that absence eventually cost physics its notion of absolute time. Right now there is a collection of very expensive null results. No supersymmetric particles. No dark matter, in detector after increasingly sensitive detector. No proton decay. No deviation in the electron’s magnetic moment. Nobody yet knows whether these are disappointments or whether one of them is the next Michelson–Morley.

In a spiral galaxy, the ratio of dark-to-light matter is about a factor of ten. That’s probably a good number for the ratio of our ignorance to knowledge.
Vera Rubinquoted in Science, 2002

1933Zwicky infers missing mass in the Coma cluster. Largely ignored.
1964CP violation found in kaons — real, and far too small to explain us.
1970sRubin and Ford: flat rotation curves, galaxy after galaxy.
1998The expansion is accelerating. Super-Kamiokande shows neutrinos oscillate.
2012The Higgs is found — and nothing unexpected comes with it.
2015Gravitational waves detected, confirming relativity where it is strongest.
2016–Successive dark matter detectors report nothing, at increasing sensitivity.
nowAbout 95% of the universe is unaccounted for by anything in this course.

One last thing, and it is the point of the whole course. What these forty-six lessons have tried to do is show where each claim came from. Not what physics says, but what was measured, by whom, and what it forced them to give up. That habit is the only part of this worth treating as permanent. The theories here will be revised, and this last act more than the rest — the field picture is ninety years old and visibly unfinished. The measurements will not be revised. Eddington’s photographic plates, Rubin’s rotation curves, the muons arriving at sea level in numbers they have no right to, a single electron held in a trap and asked how magnetic it is: those stay true. Whatever replaces the theories in this course will have to account for every one of them. That is what it means for physics to be cumulative, and it is why the answer to how do we know this? is the only question that ever really mattered here.