Three weeks ago, I saw a nice mechanical recreation of a PCA (or total least squares) on Twitter: just by pulling some strings attached to a straw.
Some days ago, Joshua Loftus published Least squares as springs, where the author presents some nice visualisations, and explains that the cost function is the same (except for a constant) as the potential energy of springs attaching the data points to the regression line. As a result, if we take any line attached in this way to a point cloud (with the required constraints in place for the strings: vertical movement for regular regression; no constraints for PCA, the springs slide freely), then the system will oscillate until it reaches the state of minimum energy (i.e., meets the regression line).
- linear regression vs. PCA;
- covariance matrix for data generation;
- number of samples;
- initial angle and shift of the center of mass;
- velocity loss and inertia (which determines the damping ratio).