One specific reason why they rolled a custom scripting language is that they needed vector/matrix types (i.e. vector3f, etc) exposed to scripts, and adding those through an interface like cpython or lua's embedding api would have produced scripts that were incredibly slow at manipulating vectors. Presumably luajit might have fixed this, but I can understand that they didn't go that far.
Torch seems to be a concrete example of a numerically oriented luajit embedding that provides high performance linear algebra primitives. http://torch.ch/
