Isn't there an attempt to do STM in OCaml already underway?
Anyway, you will have to refrain from (almost) any side-effects while inside an STM transaction. That comes naturally to Haskell --- but in OCaml they tend to be used them more often as far as I know. This requires some discipline.
http://article.gmane.org/gmane.lisp.lispworks.general/9700/m...