I never got this approach for three reasons. The first one is sure, you could say that some "cruft accumulates". But by rebooting you're guaranteeing that if something goes only slightly wrong you may not notice it and every month you'll start with a clean system that does not show the issue anymore. So you're choosing potentially ignoring tiny issues instead of letting them crash the system in a visible way and fixing them properly forever.
The second is that there shouldn't be any "cruft". Servers are not running win95 which reliably crashed given enough time to run. "Cruft" should be fixable - if it isn't then you're running a system which cannot really be supported.
The third one is that if you cannot say anything more specific than "cruft", then your system is badly managed. Are you restarting because your app leaks memory? Is it leaving zombie processes? Is it leaving dead connections to the database? Or maybe something else entirely? Restarting can be a short-term solution for some specific issue, but if it's there to remove "cruft" and "you never know" what it is, then you might as well try arranging your server room according to feng-shui or using voodoo healing to make your app run better. Either you control your system, or you don't.
> So you're choosing potentially ignoring tiny issues instead of letting them crash the system in a visible way and fixing them properly forever.
This makes the assumption that I am able to fix them. Few people realize it, but you have very little control over the system. You can easily fix configuration errors, but if the error is due to a fundamental defect in the source code of a critical package, there's nothing you can do. I do not have the time to write patches for every defect in the system, nor do I have the luxury to tolerate them. Quite a conundrum, isn't it?
> The second is that there shouldn't be any "cruft".
There's always cruft. If you plot the uptime for servers, you will find that it looks like a very steep bell curve. Very few servers run for a few days, but also, very few servers run for years at a time. Most run for a month or two, or three.
In practical terms, this means that the bugs that get fixed first are the ones that crop up immediately on boot (everyone experiences them). The bugs that get fixed next are the ones that crop up for the average user or the middle of the bell curve (server running for a month or two). The bugs that get fixed last, if at all, are the ones that crop up for the fewest users (server running for years). This article is an example of that.
So by running your server for years at a time, you are exposing yourself to a greater amount of unfixed bugs. Also, memory leaks get worse over time and even a really minor one can mean serious issues over the span of years.
> The third one is that if you cannot say anything more specific than "cruft", then your system is badly managed.
I am not restarting because there is anything wrong with the system. It is not a solution to anything. It is a preventative measure. Preventative maintenance.
First, if there is a power failure or some other problem that forces the server to reboot unexpectedly, I know it will come back up. I know that because I have designed the system to reboot regularly and I test that capability on every server, after every update.
Second, I am avoiding problems that crop up for systems that are operating outside of the average uptime.
Third, my servers boot within 1 minute at most. What's my cost for doing this? 1 minute of uptime in the middle of the night once per month? So be it.
> Few people realize it, but you have very little control over the system.
That might be the difference in our POVs... Most of the time I work in environments where we do have control over the whole system, or at least aim for it.
> If you plot the uptime for servers, you will find that it looks like a very steep bell curve.
Unless they kernel paniced, systems I took care of run from one kernel update to the other. I never experienced the "cruft" in any way.
> I am not restarting because there is anything wrong with the system. It is not a solution to anything. It is a preventative measure.
So you don't know of anything going wrong. You're not fixing anything by restarting. You're restarting just in case... it prevents something from breaking. I guess I just disagree with that reasoning.
How do you know you can restore your system to a working state in the event of an unscheduled outage, cruft or not?
You should discriminate between services and systems - make your service available 100% of the time, but you should be able to kill and restart/reload/replace systems for maintenance or other reasons at nearly any time. And you SHOULD do that, because without proof that you can do it, your DR solution is simply a best guess.
By forcing Configuration Management Software (puppet, cfengine) so that one-off fixes that get hot patched on the production server and never documented.
You can test for that in a staging environment. Restarting live services doesn't help you guarantee anything, but makes it more likely that you restart some server in a specific situation you can't recover from. (you'll never guarantee that you can recover from all situations)
The second is that there shouldn't be any "cruft". Servers are not running win95 which reliably crashed given enough time to run. "Cruft" should be fixable - if it isn't then you're running a system which cannot really be supported.
The third one is that if you cannot say anything more specific than "cruft", then your system is badly managed. Are you restarting because your app leaks memory? Is it leaving zombie processes? Is it leaving dead connections to the database? Or maybe something else entirely? Restarting can be a short-term solution for some specific issue, but if it's there to remove "cruft" and "you never know" what it is, then you might as well try arranging your server room according to feng-shui or using voodoo healing to make your app run better. Either you control your system, or you don't.