Linear-Algebra

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. The residual is the check everybody runs after a solve, and it is the one check that cannot detect the most common way a solve goes wrong. The condition number can: it converts the precision of the inputs into an error bar on the output, before any statistics are involved....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Forming X’X squares the condition number, so the formula everybody learns spends twice the digits of the two factorisations nobody is taught. Underneath, least squares is a right angle — the fit is the foot of a perpendicular onto the column space — and reading it that way explains why QR needs no inversion at all, and why the one check that defines a least-squares solution is blind to a normal-equations fit that is wrong by 321 percent....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Positive semi-definiteness is usually presented as a technical condition on a matrix. It is not: w’Sw is the variance of the portfolio w’x, so a negative eigenvalue is a combination of your variables whose variance the matrix reports as negative, and the eigenvector names it....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Every applied account of PCA triages components by the share of variance they carry, and stability does not work that way: an eigenvector is determined by the distance from its eigenvalue to the nearest other one, so the largest component in a decomposition can be the least trustworthy axis in it....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Ridge is introduced as a penalty and works as an operation on the spectrum: in the basis where the design is diagonal it multiplies each direction by s²/(s² + α) and does nothing else, which is why it shrinks hardest exactly where the data saw least....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. The diagonal of the projection matrix always sums to the number of columns, so average leverage is p/n and no data can change it — but nothing stops one row taking a whole unit....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Eckart and Young settled which rank-k matrix is closest to yours — your own truncated SVD — and said nothing about which k. The scree plot fills that gap with a judgement about a shape and no reference distribution, and on a matrix with no structure in it that judgement returns a number every time....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Logistic regression has no closed form, so it is iterated, and the iteration is exactly the weighted least squares of the previous seven episodes: Newton’s method on the log-likelihood is a reweighted normal-equations solve, and it converges quadratically when there is something to converge to....