Can you be more specific about what SPC algorithm you moved to? Did you trade off prediction quality for complexity, increasing the number of false alarms?
We generally targeted specific statistics of derived/processed streams. For some such streams we cared if the mean changed. In others if the spread changed in a way that was unusual for the time of day. In yet others if some percentile changed that was unusual for the time of day. Sometimes it will be more than one of such statistics.
Then we would track an online estimator of that measure with an SPC chart. The thresholds would be set based on our appetite for false alarms. We did not fit or use properties of parametric distributions that standard SPC charts use. So no 3-sigma business. In our case convergence to Gaussian would often be not fast enough for such techniques to be useful.
Also the original streams were far from IID, temporal dependencies were strong. So we had to derive from them derived streams that didn't show temporal dependencies any longer, at least not as strongly. This was the most important bit.
The next key aspect was to keep the alerting thresholds as untarnished and unaffected as possible from the outliers that would unavoidably occur. Getting this to work without additional human supervisory labels was the next most important part.
Make this part too robust to outliers then the system would not automatically adapt to a new normal. Make it too sensitive and we would get overwhelmed by false positives.
> So we had to derive from them derived streams that didn't show temporal dependencies any longer, at least not as strongly
Could you expand on that (or at least point me towards some keywords to read up on)?
I think I understand conceptually what this means (network latency increases at noon CDT each day because YouTube load increases during the lunch hour, as an example) but I'm wondering how you normalize data streams for temporal dependencies with unknown frequencies (nominal change #1 happens each Sunday, while nominal change #2 happens each day at noon).
That's where you need time series models. Take a look at linear filters and how to handle seasonal variations.
It gets interesting when there are different frequencies at play simultaneously, like in your example -- day of the week and time of day effects. If their periods are mutually prime that's more convenient to handle as they are orthogonal in the Fourier space.
