Iterative Local Dynamic Programming

I like a good paper. This is a good paper:

• Todorov E and Tassa Y, "Iterative local dynamic programming," in proceedings of the 2nd IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, pp 90 - 95, 2009 (pdf)

Actually, all of the papers on Emanuel Todorov's page are really worth looking over.

They have a very simple algorithm for locally optimizing the trajectory of a nonlinear stochastic system. I have used differential dynamic programming for some time, but I believe this method is simpler and faster.

The key insights are the use of linear basis functions for fast evaluation of the value function partial derivatives and the use of a fixed sample cloud of states around the nominal trajectory that is used to fit the value function and feedback controller.

I've been working on my own matlab implementation, but haven't gotten it to work just yet. It still doesn't solve one of my major criticisms of these non-parametric trajectory optimization approaches: you have to give it a good initial trajectory, which can be difficult in many cases.