> Our services now ran on Kubernetes with individual pods collecting tens-of-thousands of metric samples per second. Engineers also evolved much stricter expectations for the amount of dropped metric samples (none).
Is there any indication of how many datagrams can be dropped in a localhost connection? I expect zero datagrams to be dropped between a service and it's sidecar.
It's also possible to run a StatsD node in a Kubernetes cluster as a DaemonSet. Each service+StatsD sidecar runs as a Deployment and StatsD sidecars can be configured to both aggregate metrics and push them to the StatsD DaemonSet through a TCP connection. This would lead to very low traffic rates between each Deployment and the DaemonSet, as sidecars can push their metrics in periods of several seconds or even minutes, and each Kubernetes node typically runs single digit/low two digit Deployments which results in single-digit TPS.
Does anyone with experience in both Prometheus and StatsD have any insight into this scenario? At first glance it sounds like the premise to push a migration to Prometheus is flawed and unjustified.
> Our services now ran on Kubernetes with individual pods collecting tens-of-thousands of metric samples per second. Engineers also evolved much stricter expectations for the amount of dropped metric samples (none).
Is there any indication of how many datagrams can be dropped in a localhost connection? I expect zero datagrams to be dropped between a service and it's sidecar.
It's also possible to run a StatsD node in a Kubernetes cluster as a DaemonSet. Each service+StatsD sidecar runs as a Deployment and StatsD sidecars can be configured to both aggregate metrics and push them to the StatsD DaemonSet through a TCP connection. This would lead to very low traffic rates between each Deployment and the DaemonSet, as sidecars can push their metrics in periods of several seconds or even minutes, and each Kubernetes node typically runs single digit/low two digit Deployments which results in single-digit TPS.
Does anyone with experience in both Prometheus and StatsD have any insight into this scenario? At first glance it sounds like the premise to push a migration to Prometheus is flawed and unjustified.