is an alternative if visuals are all that matters. It can and will rain havoc in the Fourier space.

Control theory and reinforcement learning are different ways of looking at the same problem. They traditionally and culturally focussed on different aspects.

We have successfully replaced thousands of complicated deep net time series based anomaly detectors at a FANG with statistical (nonparametric, semiparametric) process control ones.

They use 3 to 4 orders lower number of trained parameters and have just enough complexity that a team of 3 or four can handle several thousands of such streams.

The amount of baby sitting that deep net models needed was astronomical, debugging and understanding what has happened quite opaque.

For small teams, with limited resources I would still heavily recommend stats based models for time series anomaly detection.

May not be your best career move right now for political reasons. Those making massive bets do not like to confront that some of their bets might not have been well placed. They may try to make it difficult for contrary evidence to become too visible.

This is one of the reasons I am so skeptical of the current AI hype cycle. There are boring, well-behaved classical solutions for many of the use-cases where fancy ML is pushed today.

You'd think that rational businesses would take the low-risk snooze-fest high-margin option any day instead of unintelligible and unreliable options that demand a lot of resources, and yet...

In 2013 my statistics professor warned that once we are in the real world, "people will come up to you trying to sell fancy machine learning models for big money, though the simple truth is that many problems can be solved better by applying straightforward statistical methods".

There has always been the ML hype, but the last couple years are a whole different level.

Say you have bet billions as a CEO, CTO, CFO. The decision has already been made. Such a steep price had to come at the cost of many groups and teams and projects in the company.

Now is not a time to water plants that offer alternatives. You will have a smoother ride choosing tools that justifies that billion dollar bet.

I think an uncomfortable reality is that a lot of decisions (technology, strategy, etc.) are not optimal or even rational, but more just an outcome of personal preferences.

Even data-driven approaches aren't immune since they depend on the analysis and interpretation of the data (which is subjective).

After all even in Physics big advances came from thought experiments. Data is one way to reason about a decision, logic and knowledgebase is another way. Both can be very powerful if one retains the humility of fallibility.

In organizations one common failure mode is that the organisational level at which decisions are made are not the same levels where the decisions are going to have their effects felt.

It's a really difficult problem to solve. Too much decentralisation is also a bad idea. You get the mess of unplanned congested cities.

This has been observed forever in various forms/contexts. Planning & policy people call them "Wicked Problems" (https://en.wikipedia.org/wiki/Wicked_problem). The Philosophy of Science one goes by the Demarcation Problem (https://en.wikipedia.org/wiki/Demarcation_problem) { roughly, in the sense that the really hard nugget connects to "trust" }.

At least one aspect of all of it is that trust is a little like money/capital and "faking it" is a bit like "stealing". The game theory of it is that since faking is virtually always vastly cheaper there are (eventually) huge incentives to do so, at some point by someone(s). So, almost any kind of trust/delegation structure has a strong pull toward "decay", from knock-off brands to whatever. It just takes a sadly small fraction of Prisoner's Dilemma defectors to ruin things/systems thereof. 2nd law of thermo makes order cost energy and this decay feels like almost an isomorphic (maybe even the same..?) thing. It's not just product/tech enshittification, but that might be yet another special case/example.

Anyway, I have no great answers or as some responder to me a while back said, if I did, I'd "have a Nobel and possibly be the first president of the united planet".

I know some examples but not too many. Care to share more examples?

- Instead of trying to get LLMs to answer user questions, write better FAQs informed by reviewing tickets submitted by customers

- Instead of RAG for anything involving business data, have some DBA write a bunch of reports that answer specific business questions

- Instead of putting some copilot chat into tools and telling users to ask it to e.g. "explain recent sales trends", make task-focused wizards and visualizations so users can answer these with hard numbers

- Instead of generating code with LLMs, write more expressive frameworks and libraries that don't require so much plumbing and boilerplate

Of course, maybe there is something I am missing, but these are just my personal observations!

We really don't. There are demos that look cool onstage, but there is a big difference between "in store good" and "at home good" in the sense that products aren't living up to their marketing during actual use.

IMO there is a lot of room to grow within the traditional approaches of "yesterday" - The problem is that large orgs get bogged down in legacy + bureaucracy, and most startups don't understand the business problems well enough to make a better solution. And I don't think that there is any technical silver bullet that can solve either of these problems (AI or otherwise)

If you're not convinced, I suggest you to search for the law firms, hospitals, and laboratories ... all of which are using AI models as of today to do both the research and boiler-plate work. Creative industries are being literally erased by the generative AI as we are speaking. What will happen with the Photoshop and other similar tools when I can create whatever I want using the free AI model in literally 2 seconds without prior knowledge? What will happen with majority of movie effect makers when single guy will be able to do the work of 5 people at the same time? Or interior designers? The heck, what will happen with the Google search - I anticipate nobody will be using it in a year or two. I already don't because it's a massive sink of time compared to what I can do with perplexity for example.

There's many many examples. You just need to have your mind open to see it.

* Show me a discrete manufacturing company using AI models for statistical process control or quality reporting

* Show me a pharmaceutical company using AI models for safety data analysis

* Show me an engineering company using AI models for structural design

The list goes on and on. There are precious few industries or companies that have replaced traditional analysis & prediction with AI. Why? Because one of two things are true: 1) their data is already in highly structured relational stores that have long legacies of SQL-based extraction and analysis, 2) they're in regulated industries and have to have audit-proof, explainable reporting, or 3) they need evidence-based design and analysis that has a key component coming from real people observing real processes in action.

For all the hyped "AI Automation" you read about, there are 100 other things that aren't, or where firms don't believe they can be, or where they'll struggle to for [reasons].

The failure cases are those where AI solutions have to stay in a continuous debug, train, update mode. Then you have to think about the resources you need, both in terms of people as well as compute to maintain such a solution.

Because of the way the world works, it's endemic nonstationarity, the debug-retrain-update is a common state of affairs even in traditional stats and ML.

Yet, we have such systems in place where we don't have to retrain the model with ever-growing data. This is one example I could think of but it kinda suggests that models, at least for some purposes, don't have to be retrained continuously to keep them running well.

I also use a technique of explaining something to the AI model he has not seen before (according to the wrong answer I got from it previously), and it manages to evolve the steps, whatever they are, so that it gives me the correct answer in the end. This also suggests that capacity of the models is larger than what they have been trained on.

There's almost always something or the other breaking. Did the nature of data change. Did my upstream data feed change. Why are these small set of examples not working for this high paying customer.

You would need resources to understand and fix these problems quarter after quarter.

A rich network of data dependencies can be a double edged sword. Rarely are upstream code and data changes benign to the output of the layer you own.

There are two cases where AI solutions are perfect. They are so good that they are fire and forget. The second is that your customer is a farmer not a gardener. Individual failing saplings mean little to him.

If a single misbehaving plant can cause commercially significant damage then when choosing opaque tools you must consider the maintenance cost you may be signing up for.

Say I have a ton of historical data that is being continuously added to. It's a real temptation to replace the raw data with a model that uses less number of parameters than the raw data. In a sense lossy compression. Can be a very bad idea. Data instances where the model does not fit well may be the most important pieces of art information. Model paints with a broad brush stroke. If you are hunting faults, you have been aware that a lossy compression can paper them away. You are also potentially harming a future model that could have been trained but you have thrown away a decade of useful data because storage costs were running so high.

No easy solution. General recommendation would be to compress but losslessly simply because you know not what may be valuable in the future. If it's impossible, then so be it, you have to eat that opportunity cost in the future, but you did your best.

Some options have more persuasive salesmen than others.

Autoscaling? Data center cooling and power use?

Interesting.

Were you using things like Matrix Profile too ? And if so, have those been replaced too ?

Among other reasons, this is largely true because acceptable ranges for different anomaly & defect types can vary significantly for different revs of a single product, or even sub-revs (things that are tied to an ECO but don't result in incrementing the product rev), or -- more crucially -- the line the product is manufactured on. One thing that's notoriously tricky to troubleshoot without being physically onsite is whether a defect is because of a machine, because of a person, or because of faulty piece parts/material.

Understanding and knowing how to apply traditional statistical analysis to these problems -- and also designing useful data structures to store all the data you're collecting -- is far more valuable right now than trying to shoehorn in an AI model to do this work.

Then we would track an online estimator of that measure with an SPC chart. The thresholds would be set based on our appetite for false alarms. We did not fit or use properties of parametric distributions that standard SPC charts use. So no 3-sigma business. In our case convergence to Gaussian would often be not fast enough for such techniques to be useful.

Also the original streams were far from IID, temporal dependencies were strong. So we had to derive from them derived streams that didn't show temporal dependencies any longer, at least not as strongly. This was the most important bit.

The next key aspect was to keep the alerting thresholds as untarnished and unaffected as possible from the outliers that would unavoidably occur. Getting this to work without additional human supervisory labels was the next most important part.

Make this part too robust to outliers then the system would not automatically adapt to a new normal. Make it too sensitive and we would get overwhelmed by false positives.

Parameter instability does not worry a machine learner as much as it worries a statistician. ML folks worry about output instabilities.

The current understanding goes that this overparameterization makes reaching good configurations easier while keeping the search algorithm as simple as stochastic gradient descent.

(Asking specifically about time series models and such.)

https://arxiv.org/abs/1803.03635 ( you can follow up on semantic scholar for more)

Selecting which weights to discard seems as hard as the original problem. But random decimation, sometimes barely informed decimation have been observed to be effective.

On the theory side now it's understood that in the thicket of weights, lurk a much much smaller subset that can have nearly the same output.

These observations are for DNNs in general. For time series specifically I don't know what the state of the art is. In general NNs are still catching up with traditional stats approaches in this domain. There are a few examples where traditional approaches have been beaten, but only a few.

One good source to watch are the M series of competitions.

I have an IoT use-case, I wanted to look both at NNs and more classical stats models to see if it has value

Look up nonparametric statistical process control and you will find useful papers. The algorithms are actually quite simple to implement. If the algorithms are not simple then probably they are not worth your time. The analysis in the paper might be complicated, don't worry about that, look for simplicity of the algorithms.

Assume you have signal from one IoT device, say a sensor reading. Anomalies are sudden changes in the value of the signal. Define sudden (using the time delta between observations and your other domain knowledge); let's say the sensor reports 1x/second and sudden means 1-3 minutes.

Simple options: rolling mean last 3 values/rolling mean last 60 values. If this value is over a threshold, alert

Say the readings are normally distributed, or they can be detrended/made normal via a simple 1 or 2 stage AR/MA model. Apply the https://en.wikipedia.org/wiki/Western_Electric_rules to detect anomalies.

Complexer but still simple options. Say you have IoT sensors over a larger area, and an anomaly is one sensor which is higher than others. Run roughly the same analysis as above, but on the correlation matrix of all the sensors. Look for rapidly changing correlations.

example: temperature detectors in each room of your house, and your kid opens the front door to go play in the snow. The entry hall cools down while the rest of the house's temp stays roughly stable. You can picture what that does to the correlation matrix.

It was little more complicated to remove temporal dependencies from the original streams and we could not rely on Gaussian behaviour. Other than that, it's pretty much the same, barring an effort to keep the alerting thresholds unaffected by recent anomalies.

From your comments it is clear that they don't. Super infuriating. Why should they care what I do with ingress and outgress that I paid for, as long as I am not hurting them.

ISPs who enforce data caps already priced it in and technically have an incentive for you to exceed your cap as fast as possible so you pay to increase said cap (they can however still slow down your traffic as they wish, to ensure sufficient capacity for everyone).

ISPs who don't enforce a cap actually still internally enforce a reasonable cap of several terabytes at their discretion. And of course, they can and will use traffic shaping to ensure the integrity of their network so your usage doesn't affect others. If you exceed that soft cap consistently several months in a row they may get in touch, but other than that you're fine.

TLDR: host your server and enjoy. When you get to the scale of the next YouTube, then you have to worry.

In my other comment I suggested carbon fibre flywheels (for energy storage). A design that stresses the rotor uniformly to near it's breaking point would make a great storage device. If it's possible to add density to the fibres but without compromising strength, even better.

For a solid material with equal strength in all direction the optimal cross section is one with an exponentially decreasing thickness.

To give an intuitive reasoning, the more radially inwards you go there's is more material and velocity on the outside that's straining to break free, so you need larger cross-section to resist that. But now, this extra thickness too has to be supported as you move inwards. One can make this formal as a differential equation and the solution is an exponential profile.

Anyhow, for carbon fibres the optimal geometry will depend on the weave because a fibre has different strength along different directions.

There was also some research for using such flywheels for energy recovery in very heavy vehicles with electric motors, e.g. tanks with a turbo-electric generator, but the use in a vehicle has obvious difficulties. Even if the flywheels are paired, to avoid influencing the mobility of the vehicle, that still causes high internal stresses in the case holding the pair of flywheels when the vehicle rotates, which can lead to fatigue failures.

I chose it as my undergraduate project literally several decades ago.

3D woven ones might be stronger as they might resist laminar separation of circumferential layers more. Going by units, the product of stress and volume has the same units as kinetic energy. So it appears breaking stress and volume might be what limits the stored kinetic energy. This addresses doubt and curiosity raised by one comment (not yours).

https://news.ycombinator.com/item?id=46039952

Wonder if I should braid my wired earphones for storage to prevent tangling. I can keep the cable inside a pouch with the earpieces out but that's not very satisfactory.

My current fascination knitted ropes/cables/cords. These are not the typical ropes that are spun and coiled and held together by friction. These ones made of synthetic fibres look like woven tubes, but the insides aren't hollow. The insides seem packed with more woven tubes.

What I really want to see though, are 3d knitted heavy duty carbon fibre flywheels of optimal shape such that it's under equal radial stress everywhere. The shape is interesting to compute for a solid one.

Regarding where can they be used... they are just batteries with a different form factor. You can put them on the grid to have some inertia and be a place to dump or extract energy spikes. They may be commercially viable if rare earth based batteries become very expensive. At a smaller scale one could use them as a mechanical UPS for a building/datacenter. Maybe even to power golf carts, not sure how well they will steer because of the angular momentum.

Even on the grid, batteries for sub-hour duration storage are cheap, as long as you place them at an already existing AC/DC converter site like a solar plant (or a modern internally-DC datacenter's centralized grid rectifier (AC/DC converter)).

Or even a HVDC transmission line.

Or a sufficiently modern aluminum/zinc smelter. Pretty much anything large enough to bother that has at least a boost-PFC on the input. Because with those you could just put them there, beef up the capacitor a bit or better yet, use a native 3-phase PFC that doesn't have strong 100/120 Hz ripple on that capacitor, and then literally just control the already there input transistors to do your grid jobs. If it was the very cheap low efficiency rectifier approach, it also needs the rectifier upgraded to be controlled, so just use it on the higher efficiency ones before upgrading the others. (It's like 1% efficiency difference on 240V, and 2% on 120V power supplies.)

There is no competition between batteries (low power density, high energy density, low storage cycle efficiency) and flywheels (high power density, low energy density, high storage cycle efficiency).

Flywheels (preferably levitated in vacuum) compete only with supercapacitors and superconducting rings (SMES = Superconducting Magnetic Energy Storage).

Supercapacitors/flywheels/SMES have their high-power applications, for which batteries are not appropriate.

Using them where batteries are the right solution is of course not a good choice.