I've been sitting on the fence wrt Terraform and other such tools for quite some time now. After being _forced_ to finally write massive k8s YAML files (and ansible YAML files) for a consulting gig, I've been wondering whether these tools should be developed as _libraries_, that you glue together using a full-fledged programming language, instead of shoe-horning a programming language in YAML.
For example, could the following be library functions that you could glue together in the programming language of your choice: (a) get current state of infra, (b) calculate diff between desired state and current state, (c) perform a single step (safely) that represents a granular change in infra, (d) perform a series of steps representing infra changes with safe rollback?
You‘re pretty much describing the idea behind Pulumi which got a lot of traction lately.
Personally, I‘m still undecided on whether the unlimited freedom of a fully fledged programming language is a good or a bad idea in terms of footgun potential.
I‘m also still a bit unsure whether to play early adopter for an extremely hyped VC open core project even though it feels tempting.
Pulumi sounds interesting. Spent 10 mins with their marketing website and I'm not very clear whether it is a standalone set of libraries, or do they only work in conjunction with their cloud services. Do you know?
I've been using Pulumi for a new project after using Terraform for a long time. It's a little weird at first, but then it clicks and actually feels quite nice. The Input/Output logic with its async behavior is the weird part, but it works fine when you understand how it works.
The only (minor) problem that I've seen in it is that the JavaScript/TypeScript support seems more mature and featureful than the other backends. So, I'll simply use that.
I'm closely tracking an effort by Microsoft that aims to do a lot of what you're describing since I find myself bridging between these tools and deploying stacks that span tools and roles. [CNAB](https://cnab.io/) and the front-running implementation, [Porter](https://porter.sh/), enable one-step infra deployments, packaged as a single OCI-compatible container, with any number of steps, using the best tools for each of those steps. Think of using aws-cli for some initialization step (create or verify presence of a state bucket), applying some terraform to create infra, and finishing with a helm chart to complete deployment of app components. Each stage in a bundle packages not only the code to run it but also the execution binary of the tool that runs it. The spec and porter are still a moving target but it's a promising space and a nice adjacent evolution of the current state of tooling.
My team does something similar to this. We write our Terraform configuration as Python literals with list comprehensions, conditional expressions, etc., then use a script to dump it to JSON which the Terraform command line can parse.
For example, could the following be library functions that you could glue together in the programming language of your choice: (a) get current state of infra, (b) calculate diff between desired state and current state, (c) perform a single step (safely) that represents a granular change in infra, (d) perform a series of steps representing infra changes with safe rollback?
Does something like this already exist?