1) I think it might be of interest for users to change the cost function at run-time. With the current design this is hardly possible since it requires to instantiate a new LevenbergMarquardt class for each cost-function. Pre-Instantiation of all possible optimizers is here a bit cumbersome. An alternative would be to define a pure virtual cost function base class. Actually, in our framework we have that even for optimizers since we also like to switch between let's say Downhill Simplex (or Nelder Mead) and LevenbergMarquardt. But that is really optional - though in our group we utilize it a lot.
2) void abort() would be nice in the optimizer's interface. There exist optimization procedures that take for longer than 5 minutes and here it makes a lot of sense to be able to terminate early. How this might be done
a) Introduce a member: Status m_status; b) Replace each 'return Running;' by 'return m_status;' c) Replace each remaining 'return <status_type>' by 'm_status = <status_type>; return m_status;'
d) Add 'void abort() { m_status = UserAsked; }
3) It seems, as if currently different functions are called for numerical differentiation vs. optimal storage vs. analytical differentiation. I am just wondering whether these special cases are not candidates for templating. These are basically policies, right? Like 'use as much mem as you want' vs. 'use minimal memory' and 'differentiate analytically' vs. 'differentiate by forword diffs'.
4) Optional: A callback function would be cool. Basically, some way of getting updates of intermediate optimization results. It oftentimes helps a lot in the course of debugging/tuning specific optimization problems because you can visualize the intermediate results.
5) Finally, I would not expose member variables directly and I am wondering whether all of those functions are required in the public interface.
In general I like the work - it's cool to have a non-linear optimization based on Eigen. :) Thanks for your all the effort.