Where does free energy collide?

The portion of a system's energy that is available to do useful work at constant temperature, equal to the total energy minus the part bound up as heat that cannot be extracted. J. W. Gibbs introduced this available energy in his 1873 A method of geometrical representation of the thermodynamic properties of substances by means of surfaces, and Hermann von Helmholtz gave the quantity its name, freie Energie (free energy), in his 1882 study of chemical processes. In chemistry it predicts whether a reaction can run on its own, since a reaction proceeds toward lower free energy. This referent is a measurable physical quantity in joules, and it is unrelated to the fitting objective of an inference algorithm or the brain theory that later borrowed the name.[1][2]

Unit

A thermodynamic potential in joules, defined for a system at a given temperature

Recovery means

Not applicable

Measured by

Computed from measured enthalpy and entropy changes, or from equilibrium constants and cell potentials in chemistry

A quantity minimized when a probability model is fit by variational inference: rather than compute an intractable distribution exactly, the method proposes a simpler distribution and tightens a bound on the evidence, and the negative of that bound is called the variational free energy. Hinton and van Camp introduced this bound for neural networks in their 1993 Keeping the neural networks simple by minimizing the description length of the weights, Hinton and Zemel named it after Helmholtz free energy in their 1994 Autoencoders, minimum description length and Helmholtz free energy, and Jordan, Ghahramani, Jaakkola, and Saul set out the general method in their 1999 An introduction to variational methods for graphical models. Minimizing it pulls the proposed distribution close to the true one while keeping the model simple. This referent is a mathematical objective computed on a machine, borrowing the thermodynamic name by analogy, and it is neither a physical energy in a chemical reaction nor a claim about how brains work.[3][4][5]

Unit

A scalar objective in nats or bits, evaluated for a proposed distribution over a model's variables

Recovery means

Not applicable

Measured by

Calculated during model fitting as the negative evidence lower bound and tracked as optimization drives it down

A theory that any self-organizing system which resists disorder, the brain included, acts to minimize its variational free energy, a mathematical bound on how surprising its sensory input is. Karl Friston, James Kilner, and Lee Harrison proposed this in their 2006 A free energy principle for the brain, and Friston developed it into a single account of perception, learning, and action in his 2010 The free-energy principle: A unified brain theory?. On this view both perception and action work to reduce the same quantity, by updating internal predictions or by acting to make the world match them. This referent is a proposed principle of brain function that reuses the inference objective from machine learning, and it is not the thermodynamic energy of Gibbs and Helmholtz nor simply the fitting objective of one algorithm.[6][7]

Unit

A scalar bound on sensory surprise, defined over an organism's internal model

Recovery means

Not applicable

Measured by

Estimated within computational models of the brain and tested against neural and behavioral data