All you can do if you don't trust the OS is to assemble the password at the point of use (each time) and erase the memory location directly afterwards.
And even that is not 100% foolproof (the OS can detect it in between these two steps).
You could store the decryption key on the disk, only loading it when needed, and possibly byte-by-byte. This is all hackable, especially when such techniques are used mainstream, but it increases the amount of work needed to hack something. In the end it's the OS's responsibility of course.
The .NET framework has a SecureString class which does this. It mitigates the risk of sensitive information being discovered through running 'strings' on memory dumps and the like. It's meant to prevent trivial recovery, not stop an attacker with sufficient time and skill.
And even that is not 100% foolproof (the OS can detect it in between these two steps).