These days I use Ansible to take care of all of that. I can’t share the scripts due to company restrictions, but it has probably built ~10,000 servers in the last year.
I first run a bunch of checks to try and make sure the build will be successful. Then create the instance where cloud-init does a few basics to allow the rest to work. Once the instance is up, the connection in Ansible flips from localhost to the server, mounts the drives, installs everything that needs to be installed and does whatever configuration is needed, and adds records in whatever systems of record need to be updated. The whole process takes about 10 minutes or so (for a single sever), depending on some external dependencies. The time increases as the count goes up. That’s probably something we could solve for, but it hasn’t been a big deal so far.
It's also handy for keeping systems in a desired state but I use it a lot more for "clone this when you boot" times hundreds of devices. Error recovery is left as an exercise to the reader but mostly a matter of configuring log egress early and having sane health check policies
I’ll have to look into this a little to see if it might be useful for some other things we have, if we’re even allowed to used it. We’re running everything through AAP which is useful for audits. We do have a home-grown tool for installing and maintaining state of a lot of the things we need to setup. We install that during setup with Ansible and wait for it to finish its initial setup to make sure there are no errors before we turn it over to the customer.
The other issue we have is our change management process. We need to provide a single update with the status of everything when it’s done. So our process can only go as fast as the slowest server. We could change this, but it would be a political nightmare I don’t want to deal with.
I first run a bunch of checks to try and make sure the build will be successful. Then create the instance where cloud-init does a few basics to allow the rest to work. Once the instance is up, the connection in Ansible flips from localhost to the server, mounts the drives, installs everything that needs to be installed and does whatever configuration is needed, and adds records in whatever systems of record need to be updated. The whole process takes about 10 minutes or so (for a single sever), depending on some external dependencies. The time increases as the count goes up. That’s probably something we could solve for, but it hasn’t been a big deal so far.