I was relatively lucky in that regard… the Manager that “owned” the clusters gave me a free hand and I worked with the users daily. Most of the software was very specialized, expensive and licensed by the seat. Installation of the software packages could take hours and a lot of it needed to be complied from source code. Once installed I had a few users that had the time to thoroughly test everything before full release to the rest of the users. All my users had extensive requirements and little to no time to spend on multiple cluster management, too.
Almost all sat on MS desktops and accessed the clusters through ssh and knew little about Linux other than the commands required to load, maintain, and run their data. I had lots of “teachable” moments and was happy to help them out. They could manage their own desktops with few issues other than security requirements and other administration requirements in a top secret environment.
Again, lucky for me, my only responsibility was to ensure their systems were properly setup to access and use my clusters as well as offsite clusters at places like Lawrence Livermore, etc. The day-to-day MS desktop system requirements were handled by a fully qualified MS Admin.
I understand your comments on management. I essentially had two different management groups that I had to satisfy, the “owner” and users of the clusters, all happy to hand over that responsibility to me, and then the separate upper level management that had no clue about computer administration or Linux. Their requirements could be onerous at times due to their ignorance of systems management which in general was minuscule. So I was able to work that to my and my users advantage. And, thanks to the owner of the clusters, I, and my users, won far more battles than we lost.
Every location is different and often has specialized requirements that are unique to the job. And that’s why I feel that AI is very limited in systems management capability, the subject of this thread.
Generic AI stuff is fine if the administrator has no other tools such as splunk, nagios, Grafana/Prometheus, cockpit, and general command line tools (btop, netcat, etc) that can give far more detailed troubleshooting and adminitrative real-time info than generalized AI agents.
Particularly in assisting both hardware and software root cause analysis when trouble happens, and it often does. Good systems administrators with skills in hardware and network setup and maintenance, programming and compilation skills, and OS management will always be needed for large installations.
