> Timing, performance analysis, and CPU/memory availability are huge ones. Being resource constrained in a VM is not an acceptable approximation of what a user's desktop actually is.
Fair enough. Don't you run into the same problem with volunteer machines though? You can't actually control their configuration in TestSwarm so you can't control the resources (e.g. you can't control what else the machine is doing). If you are going to do do any serious analysis of timing or performance then you are going to have to control the machines.
> If you are going to do do any serious analysis of timing or performance then you are going to have to control the machines.
Absolutely - the timing and performance won't be more accurate but it will be more realistic. For accuracy everything has to be done on the same machine in a 'clean' environment (and definitely not in a VM).
Fair enough. Don't you run into the same problem with volunteer machines though? You can't actually control their configuration in TestSwarm so you can't control the resources (e.g. you can't control what else the machine is doing). If you are going to do do any serious analysis of timing or performance then you are going to have to control the machines.