The question isn’t whether there’s some way to do arithmetic on raw pointers, but whether you can take a pointer to a subobject and expect the GC to keep the object alive for you. In Go you can do this, but you can’t in Java or C#, I believe.
Although it isn't visible from the language level the JVM internally does have "derived pointers" which are normally stored on the stack and point to an offset within an object. The GC can figure this out because it has a map for what the stack entries mean.
'The' JVM can internally have whatever it likes; so long as it is not exposed at the language level, nothing prevents an implementation from being devised which does not have such things internally, and which has a treadmill.
Java used to do something like that for strings, where taking a substring of a string created an object that pointed into the original string, but that only was for strings, and didn’t expose the pointer, so the implementation could keep a pointer to the original string around for the GC to use.
I bet there isn't a language around (with GC) that tops their GC implementations.