Sure, as long as you are working with a standard and linear memory model. The assumption that size_t and ptrdiff_t can store a pointer may break down on segmented memory architectures for example. If you want to code against the standard instead of an assumption, use uintptr_t and intptr_t instead.
When would someone want to use intptr_t instead of uintptr_t? Does a signed memory address even make sense? Perhaps intptr_t is available to avoid compiler warnings about mixing signed/unsigned ints when adding a uintptr_t and a (signed) ptrdiff_t.
While I don't know of any specific use-case, it is of course very easy to imagine an architecture with signed memory-address space. One could for example separate protected/kernel memory from user land memory with the signedness.
