What I’ve seen so far is a lot of people basically getting good starts with something AI but mostly still having to correct too much so that either the prompt takes more work than the actual task or the work thereafter is more.
And I have seen some situations where llm also really made sense and was helping in general, but that was far and in between.
Anthropic enterprise models are very strong at following runbooks and managing infrastructure. To frame this properly, I spent about a month documenting the environment and creating runbooks. I can currently do things like the following:
Follow the steps below to roll over this cluster. {
Choose a server in this cluster by ID in numeric order (this server)
Use Ansible+vault to create a backup AMI of (this server)
Use Ansible+vault to collect health data on (this server) according to the server health runbook
Use Ansible+vault to disable notification for this server
Use Ansible+vault to preform a full kernel update and reboot the vm according to the update runbook.
Use Ansible+vault to collect health data on this server according to the server health runbook post update
Compare the pre and post health and server alarms for (this server), if there is an issue, pause, SOS notify oncall
Use Ansible+vault to enable notification for this server
}
Each runbook contains exact steps required, how to verify success, each one of them is able to pull the E-stop, freeze on that step and SOS notify.
You’re absolutely correct, 100% It’s entirely scriptable. And I have done that work many, many times in the past 15 years
The interesting differences:
I didn’t write the ansible jobs, it did to my specifications, i just had to review them. It can generate an Ansible job that would take me 30 minutes to properly write/document in less time than it takes me to review its output.
It knows how to do AWS AMI and RDS images without details. If Amazon changes something and it errors making the ami, it will, without intervention, find amazon docs and update it’s process.
My playbooks are essentially Word documents (md tho) telling it to make an AMI backup and disable notifications using the monitor server who’s name and IP are documented the neo4j. I don’t tell it where the logs are, it knows. It’ll fire up a Cloudtrail to hunt something down if it needs it.
The diffs on the health checks aren’t binary. If it sees something it can’t explain, it will go back further in time, inspect other nodes, even start going down the debugging route for the app, (as long as I give it docs on how it should debug). It doesn’t miss a line. It doesn’t blindly ignore things. I was doing a rebuild on a server the other day, and during the post-health check, it noted that a certain error on this node is present on 5 other nodes and that the most common cause is a minor config issue. I asked it to diagnose; it found a config file from the old image that we never touch. I’ve been running with that issue on 6 servers for years. By my hand alone, I’d never have gone back, researched it and fixed it. It’s truly trivial, but it’s also things like that I don’t have the time to invest in that make this so useful.
And it goes through this level of detail for every node. It only bugs me if it finds something outside of scope that’s unexplainable.
I don’t give it any passwords or hard config. It queries Amazon or Neo4j, which are kept up to date programmatically.
The real terror is I can do this on several different projects at once. As my documentation, guardrails, and automation get better, I’m getting multiple times my own hours’ work done every day, and I was really big on automation and self-healing stuff before this ever came along.
Could the node classifier go bonkers and wipe out everything? It could. But I’m careful not to just cron up a bunch of jobs and take a nap. Probably the next step would be to make some overseers to verify backups are being made and it’s waiting between jobs.
Could you tell which field that job is in?
What I’ve seen so far is a lot of people basically getting good starts with something AI but mostly still having to correct too much so that either the prompt takes more work than the actual task or the work thereafter is more.
And I have seen some situations where llm also really made sense and was helping in general, but that was far and in between.
Anthropic enterprise models are very strong at following runbooks and managing infrastructure. To frame this properly, I spent about a month documenting the environment and creating runbooks. I can currently do things like the following:
Follow the steps below to roll over this cluster. {
Choose a server in this cluster by ID in numeric order (this server)
Use Ansible+vault to create a backup AMI of (this server)
Use Ansible+vault to collect health data on (this server) according to the server health runbook
Use Ansible+vault to disable notification for this server
Use Ansible+vault to preform a full kernel update and reboot the vm according to the update runbook.
Use Ansible+vault to collect health data on this server according to the server health runbook post update
Compare the pre and post health and server alarms for (this server), if there is an issue, pause, SOS notify oncall
Use Ansible+vault to enable notification for this server
}
Each runbook contains exact steps required, how to verify success, each one of them is able to pull the E-stop, freeze on that step and SOS notify.
Ah, thank you for elaborating!
Though I have to say, this seems to be a thing I also created tooling for in the past.
So what I did for clusters was create an optional script where servers could drop a healthcheck if they’re member of a cluster.
The idea was that they are able to check their cluster status, do actions and check if they’re OK again.
So basically a simple framework for what you’ve described and that also was long before llm.
I’m not saying this to discredit, but to add more perspectives.
You’re absolutely correct, 100% It’s entirely scriptable. And I have done that work many, many times in the past 15 years
The interesting differences:
I didn’t write the ansible jobs, it did to my specifications, i just had to review them. It can generate an Ansible job that would take me 30 minutes to properly write/document in less time than it takes me to review its output.
It knows how to do AWS AMI and RDS images without details. If Amazon changes something and it errors making the ami, it will, without intervention, find amazon docs and update it’s process.
My playbooks are essentially Word documents (md tho) telling it to make an AMI backup and disable notifications using the monitor server who’s name and IP are documented the neo4j. I don’t tell it where the logs are, it knows. It’ll fire up a Cloudtrail to hunt something down if it needs it.
The diffs on the health checks aren’t binary. If it sees something it can’t explain, it will go back further in time, inspect other nodes, even start going down the debugging route for the app, (as long as I give it docs on how it should debug). It doesn’t miss a line. It doesn’t blindly ignore things. I was doing a rebuild on a server the other day, and during the post-health check, it noted that a certain error on this node is present on 5 other nodes and that the most common cause is a minor config issue. I asked it to diagnose; it found a config file from the old image that we never touch. I’ve been running with that issue on 6 servers for years. By my hand alone, I’d never have gone back, researched it and fixed it. It’s truly trivial, but it’s also things like that I don’t have the time to invest in that make this so useful.
And it goes through this level of detail for every node. It only bugs me if it finds something outside of scope that’s unexplainable.
I don’t give it any passwords or hard config. It queries Amazon or Neo4j, which are kept up to date programmatically.
The real terror is I can do this on several different projects at once. As my documentation, guardrails, and automation get better, I’m getting multiple times my own hours’ work done every day, and I was really big on automation and self-healing stuff before this ever came along.
Could the node classifier go bonkers and wipe out everything? It could. But I’m careful not to just cron up a bunch of jobs and take a nap. Probably the next step would be to make some overseers to verify backups are being made and it’s waiting between jobs.
Thank you for providing more context and substance for your use cases since it’s something I have not experienced.
Keeping an eye on changes especially is always a pain.