Xjyuk

We know that loss (error) minimization originated with Gauss in the early 18th c.

More recently Friedman* extolled its virtues for use in predictive modeling:

“The aim of regression analysis is to use the data to construct a
function f(x) that can serve as a reasonable approximation of f(x)
over the domain D of interest. The notion of reasonableness depends on
the purpose for which the approximation is to be used. In nearly all
applications however accuracy is important...if the sole purpose of
the regression analysis is to obtain a rule for predicting future
values of the response y, given values for the covariates (x1,...,xn),
then accuracy is the only important virtue of the model...”

But is accuracy the only important virtue of a model?

My question concerns whether or not loss minimization is the only way to evaluate a regression function whether OLS, ML, gradient descent, quadratic or least absolute deviation, whatever.

Clearly, it's possible to evaluate multiple metrics beyond loss a posteriori, i.e., after models have been built. But are there multivariate functions which evaluate multiple metrics a priori or during the optimization phase?

For example, suppose one wanted to use a regression function which optimized fit based not just on loss minimization but which also maximized nonlinear dependence or Shannon entropy?

Are such functions available and/or have these issues been research

*Friedman, J.H., Multivariate Adaptive Regression Splines, The Annals of Statistics, 1991, vol. 19,No. 1 (March), pp. 1-67.

But is accuracy the only important virtue of a model?

The practical aspects of what a model's for is too nuanced for a theoretical discussion. Interpretation and generalizability come to mind. "Who will use this model?" should be a top line question in all statistical analyses.

Friedman's statement is defensible in a classical statistics framework: we have no fundamental reason to object to black-box prediction. If you want a Y-hat that's going to be very close to the Y you observe in the future, then "build your model as big as a house" as John Tukey would say.

This does not excuse overfitting, unless your question is ill defined [plural "you" being the proverbial statistician]. Overfitting is too often the result of analysts who encounter data rather than approach it. By going through hundreds of models and picking "the best", you are prone to build models that lack generality. We see it all the time.

Inadvertantly you also ask a different question: "Alternatives to minimizing loss in regression". Minimax estimators, estimators that minimize a loss function, are a subset of the class of method of moments estimators: estimators that give a zero to an estimating function. The goal of the MOMs is to find an unbiased estimator, rather than one that minimizes a loss.

Rational choice theory says that any rational preference can be modeled with a utility function.

Therefore any (rational) decision process can be encoded in a loss function and posed as an optimization problem.

For example, L1 and L2 regularization can be viewed as encoding a preference for smaller parameters or more parsimonious models into the loss function. Any preference can be similarly encoded, assuming it's not irrational.

For example, suppose one wanted to use a regression function which optimized fit based not just on loss minimization but which also maximized nonlinear dependence or Shannon entropy?

Then you would adjust your utility function to include a term penalizing those things, just as we did for L1/L2 regularization.

Now, this might make the problem computationally intractable; for example, 0/1 loss is known to result in an NP-hard problem. In general, people prefer to study tractable problems so you won't find much off-the-shelf software that does this, but nothing is stopping you from writing down such a function and applying some sufficiently generalized optimizer to it.

If you retort that you have a preference in mind which cannot be modeled by a loss function, even in principle, then all I can say is that such a preference is irrational. Don't blame me, that's just modus tollens from the above theorem. You are free to have such a preference (there is good empirical evidence that most preferences that people actually hold are irrational in one way or another) but you will not find much literature on solving such problems in a formal regression context.