For one startup, the MVP included deploying some Linux-based overseas factory stations, under initially impossible conditions.
For reasons that were really good in context, the software aspects of our station builds were based on a set of step-by-step written manual instructions, to configure the Linux distro to "bootstrapping" state for running the huge configuration shell script.
The huge shell script was written to be idempotent. So it could do both the initial heavy configuration and then the same or later versions of it re-run to make any engineering-change adjustments. Including re-run safely remotely, after the machine was deployed on the other side of the globe, in an active production line. If you're comfortable with Bash or Python, a script like this can be easier than shoehorning the problem into the structure of some declarative tool.
(The really good reasons would sound crazy out of context, and you wouldn't think it worked, but we did ultimately have perfect software uptime. And, yes, there were plans (before the Covid chill on our customers and VC) to do a lot of things differently in our next-gen station, including putting most of it in a container.)
Besides bespoke scripts (Bash, Python, Make), I've also used declarative off-the-shelf tools that do a lot of heavy lifting, including doing things with Terraform. (For example, Terraform to allocate a set of AWS EC2, storage, and networking, to simulate a real metal server environment, for developing some infrastructure software.) But, even with those tools, there's always some kind of documentation of it, and often also bespoke scripts.
Regarding remembering all the steps and getting them right... One of the tricks to documentation for things like "pets" servers is to have a canonical internal wiki into which pretty much all useful info that's not otherwise in Git goes. For example, if you go to the AWS Console to add or change something, have the right wiki page open, and update it, including copy&pastes, as you go. (Also, cross-link that with any issue-tracking for the task.) If you keep wiki access low-friction and high-value, it should save you at least as much time as it costs, and be even more valuable to others. (Put loosely, I've seen situations in which a project or a person's job hinges on whether or not one key sentence was captured in a wiki.)
Sometimes, this documentation can later be turned into a script.
Following these lightweight doc conventions, even my personal laptop has a wiki page on how to reconstruct its configuration, and it's kept up-to-date for years.
For reasons that were really good in context, the software aspects of our station builds were based on a set of step-by-step written manual instructions, to configure the Linux distro to "bootstrapping" state for running the huge configuration shell script.
The huge shell script was written to be idempotent. So it could do both the initial heavy configuration and then the same or later versions of it re-run to make any engineering-change adjustments. Including re-run safely remotely, after the machine was deployed on the other side of the globe, in an active production line. If you're comfortable with Bash or Python, a script like this can be easier than shoehorning the problem into the structure of some declarative tool.
(The really good reasons would sound crazy out of context, and you wouldn't think it worked, but we did ultimately have perfect software uptime. And, yes, there were plans (before the Covid chill on our customers and VC) to do a lot of things differently in our next-gen station, including putting most of it in a container.)
Besides bespoke scripts (Bash, Python, Make), I've also used declarative off-the-shelf tools that do a lot of heavy lifting, including doing things with Terraform. (For example, Terraform to allocate a set of AWS EC2, storage, and networking, to simulate a real metal server environment, for developing some infrastructure software.) But, even with those tools, there's always some kind of documentation of it, and often also bespoke scripts.
Regarding remembering all the steps and getting them right... One of the tricks to documentation for things like "pets" servers is to have a canonical internal wiki into which pretty much all useful info that's not otherwise in Git goes. For example, if you go to the AWS Console to add or change something, have the right wiki page open, and update it, including copy&pastes, as you go. (Also, cross-link that with any issue-tracking for the task.) If you keep wiki access low-friction and high-value, it should save you at least as much time as it costs, and be even more valuable to others. (Put loosely, I've seen situations in which a project or a person's job hinges on whether or not one key sentence was captured in a wiki.)
Sometimes, this documentation can later be turned into a script.
Following these lightweight doc conventions, even my personal laptop has a wiki page on how to reconstruct its configuration, and it's kept up-to-date for years.