> Self modifying code is handled with MMU traps on the writes, and invalidation of the relevant traces. It is very much a slow path though. Ideally heavy self modfying code is able to stay in the interpreter though and not thrash in and out of the compiler.
This might end up having a bad time running JavaScript VM JITed code, which self-modifies a lot.
Yeah, nesting JITs was kind of always an Achilles heel of this kind of architecture.
IIRC, they had a research project to look at shipping a custom JVM that compiled straight to their internal ISA to skip the impedance mismatch between two JITs. JITed JS (or really any extremely dynamic code that also asks for high perf) probably wasn't even on their radar given the era with even the SmallTalk VM that HotSpot derived from being a strongly typed derivative of SmallTalk.
This might end up having a bad time running JavaScript VM JITed code, which self-modifies a lot.
But all of that makes sense! Thanks!