Mostly, I am looking for a review from an API perspective; also, I need to support sparse matrices at some point but don't have any experience with them; perhaps Gael could comment here as he is the sparse expert. In particular, I will need to support sparse Jacobians and solving sparse normal equations..
I also have an implementation of Powell's dogleg method, which can be much faster than LM: