So what is the meaning of "ML experts" if all they do is trial and loss experiments! Is Math PhD just used for hiring signal rather than actual requirements to do ML projects?
It’s an exciting time because the field is getting to the point where ‘complexity is unbounded’
I.e. in many materials or chemistry fields where you start operating with over 100 variables, you begin to develop a dark arts of understanding because what is being attempted is beyond the ability of computers to model.
You can do chemistry without a PhD, but your ability to systematically try to address the complexity may be hindered without the training. Likewise with ‘ML experts’ (I hope they get a cool word some day to describe their profession).
the term on the right is the same constant so in total you need 100+1 forward function evaluations. And theres the issue of precision for small differences (mantissa).
One of the main reasons machine learning took off is because of the mathematical realization (Automatic/Algorithmic Differentiation) on how a 1 forward and 1 backward pass is more mathematically rigorous (calculates gradient vs finite differences) and much more efficient.
With the blackboxing of the algorithms, many endusers of the ML libraries end up using ML when they don't know the functional form of a map, but will refuse to apply automatic differentiation of a known complex function with large number of parameters. In contrast those endusers that made sure to understand Automatic Differentiation as a tool orthogonal to arbitrary function approximation (i.e. everyone who realizes the math part of ML is very important) will be able to apply AD (or any other tricks learnt through a mathematical perspective) in situatins where there is no need for arbitrary function approximation...
EDIT: woops I thought you were arguing for blackboxing, against mathematical interpretation upvoted
May be Ml is an empirical subject more than a theoretical subject. It is more biology than physics. More astronomy ... even is the subject is created does not meant it follows rules.
After all if intelligence comes out artificially, I hope it does not have rule.
Great point ML takes much of its inspiration from simulating brains. Therefore studying such artificial brains is much like studying biology and thus to a large degree empirical I would assume.
Yes, there is no need to have a PhD to do ML, either applied or research. Not to say there isn't good research using advanced mathematical/statistical methods, but most of it is not. It's entirely a signaling game.
