> 1b) Testing was not skewed by faster cores. CPU performance analysis was based on SPEC CPU 2006 using base rate runtime. SPEC CPU is a multi-core benchmark with metrics that scale linearly on number of CPU cores.
I wasn't suggesting that the testing was skewed by faster cores, but that your criteria favour hardware configurations optimized for fewer, faster cores because of your comparisons between packages of the same core count.
> 1c) The value analysis (based on CPU performance and price) highlights differences in pricing between services. I believe matching CPU cores and deriving value is preferable to comparing compute instances based on price.
We'll have to agree to disagree on this point. It's been my experience that in real-world situations when comparing options, nobody is going to reject higher core counts when at the same price. Generally, a decision maker looks for the lowest price that meets all requirements, or the best option that still fits within budget. They're not going to simply say that since one provider offers a package with this particular core count, that they'll only compare against other providers' options with the exact same core count regardless of price. That's what I mean by the core selections being arbitrary.
For example, an E5 2643v2 6x3.5GHz CPU costs about the same as an E5 2670v2 10x2.5GHz CPU. The 2670v2 offers approximately 25% more aggregate performance, and is clearly the better option except for the unusual cases where single-threaded performance is more important than aggregate performance. However, given how you choose to compare different packages with the same core counts, infrastructure using the 2643v2 would be favoured in your testing. The decision to compare a single 2643v2 core vs a single 2670v2 core, would be completely arbitrary.
> 2) The intent of the post was to provide and compare multiple relevant performance characteristics for these types of workloads: CPU, memory, (non-cached) storage IO, network. Each of these characteristics is analyzed separately using relevant benchmarks and runtime settings. If your workload relies more heavily on cached IO, then more emphasis could be placed on the memory performance analysis.
My point is, the vast majority of web and database servers either rely heavily on cached IO, or should be. SSD's should not be necessary for web servers; you will be able to serve a higher number of requests with the same budget using a larger amount of RAM instead. Likewise, you should fit your entire database into RAM if you can, and just resort to SSD's when you can't. And this isn't limited to I/O, you can reduce the amount of CPU resources required by caching to RAM also. As such, doing a comparison of packages based on the amount of RAM included is going to be more meaningful than doing a comparison based on the number of cores.
I wasn't suggesting that the testing was skewed by faster cores, but that your criteria favour hardware configurations optimized for fewer, faster cores because of your comparisons between packages of the same core count.
> 1c) The value analysis (based on CPU performance and price) highlights differences in pricing between services. I believe matching CPU cores and deriving value is preferable to comparing compute instances based on price.
We'll have to agree to disagree on this point. It's been my experience that in real-world situations when comparing options, nobody is going to reject higher core counts when at the same price. Generally, a decision maker looks for the lowest price that meets all requirements, or the best option that still fits within budget. They're not going to simply say that since one provider offers a package with this particular core count, that they'll only compare against other providers' options with the exact same core count regardless of price. That's what I mean by the core selections being arbitrary.
For example, an E5 2643v2 6x3.5GHz CPU costs about the same as an E5 2670v2 10x2.5GHz CPU. The 2670v2 offers approximately 25% more aggregate performance, and is clearly the better option except for the unusual cases where single-threaded performance is more important than aggregate performance. However, given how you choose to compare different packages with the same core counts, infrastructure using the 2643v2 would be favoured in your testing. The decision to compare a single 2643v2 core vs a single 2670v2 core, would be completely arbitrary.
> 2) The intent of the post was to provide and compare multiple relevant performance characteristics for these types of workloads: CPU, memory, (non-cached) storage IO, network. Each of these characteristics is analyzed separately using relevant benchmarks and runtime settings. If your workload relies more heavily on cached IO, then more emphasis could be placed on the memory performance analysis.
My point is, the vast majority of web and database servers either rely heavily on cached IO, or should be. SSD's should not be necessary for web servers; you will be able to serve a higher number of requests with the same budget using a larger amount of RAM instead. Likewise, you should fit your entire database into RAM if you can, and just resort to SSD's when you can't. And this isn't limited to I/O, you can reduce the amount of CPU resources required by caching to RAM also. As such, doing a comparison of packages based on the amount of RAM included is going to be more meaningful than doing a comparison based on the number of cores.