IT vs DevOps

I’ve had the pleasure of working on a few different teams during my career, all with different “configurations”. I’ve been on a two-person web team, where we were responsible for everything from project initiation to server deployment ourselves. I’ve been on a 20+ developer project, where my feature was just one of many fit into a large point of sale application. I’ve been on a “startup” team, where practically everyone was a new hire and we were making a new product from scratch. Throughout all this experience, one thing that has always changed has been the deployment & support story.

When my teammate and I were responsible for our own production servers, we were given full access to a live MSSQL database and Windows server with IIS. We were responsible for deploying database changes, configuring the web server, and pushing out application updates. And we sucked at it.

Sure, it worked, but often after hours of fiddling with settings and trying to figure out why it “worked on my machine” and not in production. We were not MSSQL server pros, or IIS pros (although we got pretty good at it). If anything slowed to a crawl or became unresponsive, it was up to us to remote into the production server and figure out what was wrong. And a lot of the time that involved just killing a worker process to get rid of that one blocking request that was taking minutes to execute. This was not a good setup.

When I was on the 20+ developer project, we had a separate “IT support staff”. This was great – I no longer had to worry about what server to deploy to, or when to push out updates, or any of the deployment mumbo-jumbo. But soon I realized I was a lot more constrained than I was used to. If I all of a sudden wanted to use a new ORM, or wanted to play with a new IIS handler, I all of a sudden had to ask permission.

Now, I understand why this might be the case – this separate team of IT pros have a certain configuration on all these servers, with backups and scripts and images and all the wonderful stuff that comes with a well-managed server infrastructure. But going from being able to play with and deploy whatever we wanted to address a requirement to having to justify the changes I’ve made to the system was a big step for me, and one I don’t necessarily like.

When I got the chance to work with the “startup” team, we were basically on our own. We were making a new product the company hadn’t made before, and all the setup and configuration and everything was up to us. Kind of similar to when we were our own IIS pros. But this time it was different. We had a few team members step into a “DevOps” role, which was a new concept for me. This person set up our build server, our deployment environments, our unit tests, and everything that pertained to the deployment of our applications. And he did it with code.

This to me was the best of both worlds. We had a team member, who as being part of the team knew exactly what we were doing with what technology. So if we needed to deploy a new tech, or roll out a new server, it was a simple matter of coordinating with our DevOps guy in the daily scrum meeting. But the other benefit of this was we also knew exactly what he was doing to perform these deployments. The rake scripts he used to push to each environment were committed to our source control system. Anyone (with permissions) could technically roll out a new build with a simple command line argument. And anyone could see by looking at the code exactly what was involved.

I see this as one of the disadvantages of the separate IT group. I have no idea what those guys do. I don’t know how our server farm is set up, I have no idea what version our MS-SQL servers are running, I have no idea if we’re running IIS 6 or 7 in production. A lot of times if there’s a problem in an environment, we let IT know, and they reply ‘fixed’. No reason why stuff was broken, or what we could’ve done to fix or prevent it. It’s all a big black box that they hold the key to.

And you could argue that I don’t need to know, that it’s “abstracted away” like a data access layer would be. But conversely, the IT group has no idea what my team is doing – they don’t know what problems we’re trying to solve in development, why we’re using the modules and libraries we are, and why an existing data-center tech might not necessarily meet our needs. If we need to do something different from what they’re set up for, we have to “justify” it to them, even if we’ve spent a bunch of development time working with the new tech and have proven it out on our own.

When TDD & BDD became “the thing to do”, there was a big push to move QA staff out of their traditional role of “test when it’s done” to more of a proactive “write the test plans with the dev team”. I would love to see IT move to the same setup. There’s absolutely no reason why a member of the IT team couldn’t be assigned to a development project, and join us in our daily scrum. That way, they know exactly what’s going on in the project they’ll have to deploy and manage, and we know exactly what we need to get ready to make that deployment a smooth transition.