Across the world, companies from every industry are combining their development teams and operations teams to create one unified DevOps team. Teams are increasing productivity, transparency, and efficiency, while simultaneously decreasing bottlenecks and communication silos. Whether it’s a small restaurant POS startup out of Boston, or a world renowned ad agency, companies are making the switch and reaping the rewards.
For those new to the industry, or seasoned veterans new the DevOps philosophy, the thought can be intimidating. Combining two different phases of the software development life cycle together isn’t always easy, although the results can be staggering and the metrics are easy to measure.
At its core, DevOps is combining development and operations into one cohesive and continuous process. By doing this, both teams are now on the same page at all times. This breaks down barriers and eliminates department bottlenecks.
We wanted to look at real world examples of DevOps being implemented and how it positively impacted a company. So naturally, we took to Reddit. We set out looking for a diverse group of IT professionals who would be willing to share their DevOps success stories. We asked them three basic questions, and allowed them to elaborate as much or as little as they’d like. The questions were:
Ryan works as a DevOps Specialist for Scotiabank in Canada focusing on SecOps challenges. In Ryan’s 15 years working in Information Technology, he has always had a keen eye and interest for security. The better part of his career was spent as a System Administrator, but around 2015 he made the switch from pure systems administration to DevOps… and hasn’t looked back since.
Ryan’s team is dedicated to transforming Scotiabank into Canada’s first true “digital” bank. They do this by updating not only technologies used within the bank, but also processes. They also focus on building close-knit relationships with various financial technology (“fintech”) startups across Canada and other countries where Scotiabank operates.
“It’s long been my opinion that DevOps is, at its core, a mentality… the product of people becoming frustrated with silos, with this notion of “well, it’s not my job, so I don’t give a damn.”
For me, DevSecOps (or SecOps, as I much prefer to call it) is, simply put, making sure that security is a vital component within our SDLC. We do this by not only by ensuring that the integration and/or automation of security practices and procedures are as seamless as possible, but also by keeping a constant, open dialog between developers, DevOps, and security teams.
The benefits of implementing SecOps are countless. From a technical standpoint, you are not only smoothing out serious speed bumps that affect your ability to release new code in an expedient manner, but you are also ensuring that what code does go out to production is properly secure. From a business standpoint, this improves your ability to mitigate possible reputational risks, as you’ll be able to catch bugs and vulnerabilities earlier, faster, and more frequently.
Now, I’ve talked a bit about how SecOps can help automate security processes and procedures. Honestly, automation is only one piece of the sometimes muddled, nebulous concept we have come to call DevOps. It’s long been my opinion that DevOps is, at its core, a mentality. DevOps was the product of people becoming frustrated with silos, with this notion of “well, it’s not my job, so I don’t give a damn.” DevOps, more than anything else, was meant to help break down barriers and foster relationships between developers and the people who maintain the platforms that developers use and deploy their code onto.
The same can be said of SecOps. We’re not just here to automate the hand that slaps you for writing insecure code. We want to help developers and operations people understand why you shouldn’t use weak crypto. We want to help developers understand why you should always validate and sanitize data inputs. We want to help operations teams understand why simply securing your edge is inadequate. We want to help teams understand these things, and then show them how to improve. At the end of the day, our core goal is to foster a better understanding of (and, if we’re really lucky, a passion for) security, and in so doing, improve the relationships and culture between DevOps, developers, and security teams.
I think the biggest challenge with implementing SecOps is that it’s very easy to adopt the mindset of, “well, hackers are all smarter than me, so anything I do to try and secure my application and/or platform is pointless.” This has been called “security nihilism”, and it is absolutely insidious and toxic to any software team. This kind of thinking needs to be heartily discouraged from the outset, as it can spread like wildfire and make implementing new security practices far more challenging and far less feasible. Yes, security is hard. Hackers seem like magical scary wizards who want to ruin your day. But it doesn’t have to be this way. Step up, take an interest. If they can learn these things, then so can you.
Get everyone on the same wavelength. If you can, get everyone in the same physical room and get them talking. DevOps is about constant communication and transparency. Work towards that, and the rest will fall into place.
Ali comes from the Ops side of the fence and currently works as a Lead DevOps Engineer for the online recruitment marketing leader, VONQ. For 10+ years, Ali has been as a 3rd line/senior level engineer and worked on many platforms varying in size and traffic.
“It’s a refreshing change when a ticket comes through, accompanied by an exhausted list of avenues they’ve explored…”
Appreciation for the whole journey of software development, from brain to production. The developers can also understand more of the underlying infrastructure – so there’s a fair balance between effort in optimizing code rather than adding more powerful hardware. A positive knock-on effect is a reduction in blame, as everyone can see the whole path. Even the finance department is happy, as the DevOps team can justify their additional expense. This leads to more trust and making future purchases a smoother process.
The appreciation goes further, as the developers can see where their code impacts the application as a whole. I had one colleague (front-end developer) who was very focused on “improving performance”. He spent much of his time tweaking the code, squeezing every last drop of performance out of the React application. Unfortunately, his optimizations made very little real-world impact, since the bottlenecks were in the back-end. I like to use the analogy, “there’s little point in tweaking the aerodynamics of a sports car for 200+ mph, if the engine is currently limited to 60 mph”. He realized that his weeks of tweaking made such a marginal difference and his time could have been spent much more efficiently elsewhere.
Another side effect is faster debugging, during both development and when production is on fire. Enhancing the Ops knowledge of the developers means the support ticket queue is greatly reduced, as they have already done more troubleshooting than they previously would have. It’s a refreshing change when a ticket comes through, accompanied by an exhausted list of avenues they’ve explored. That gives me a more focused start point for debugging – the result is a faster resolution.
Convincing people DevOps is not a magic pill that will change everything overnight. Tooling has been created to enhance the workflow of developers, but they have to actually embrace it. The responsibility of the DevOps engineer is to put the framework in place, not to do everything! A perfect example is containers. There’s no point in using them if the developers don’t start using and tagging images themselves (assuming the organization permits such a thing). There’s no point in introducing things like Jenkins Pipeline (via Jenkinsfile), if it’s only one person on the team (usually ‘DevOps Engineer’) who maintains them – or you’ll have a serious bottleneck – and zero shared knowledge (the “bus” factor).
The other challenge is changing people’s perspective on what development in2018 entails. Every developer we interview gets grilled on containers, networking, OS security, etc. Essentially we’re finding out how strong their Ops stuff is, as they will be using it on a daily basis. In the same way a Systems Administrator who can’t code gets left behind, so does a developer who doesn’t have any Ops.
Above all, the largest challenge I have faced is the historical work culture of some colleagues. I would argue that DevOps requires a high degree of autonomy and initiative, due to being granted more freedom. No longer do you write a quick support ticket when something breaks, and just “blame Ops” (aka “it’s your problem now”) until it’s fixed. You’re required to do extensive debugging yourself , and then submit a ticket showing the investigation. This is where it falls down – they don’t know where to start investigating outside of their small box. Despite working with some colleagues for over 12 months, they still see Dev and Ops as “us and them”.
Frustratingly, I have found the most effective method of working with these people is to leave it broken for them to eventually fix. When they complain it’s not getting fixed, and their work stops – the answer to your next question is the solution I have found most effective. If their boss is onboard with implementing DevOps, the developer will have to explain what troubleshooting they have done to resolve the issue, therefore putting the responsibility back on them. Without support from management, only you will get the blame – by both the developer, and their respective manager.
Get full support, in writing if necessary, from management. Convincing an ‘old timer’ to change their productivity workflow, in line with DevOps, is close to impossible without their manager asking/instructing them to do so. Without support, you’ll get nothing but resistance, and no actual power to change things. Make a business case to management, since it will have an impact on productivity and expenses. If they agree, go for it!
Yaron is a DevOps engineer with a vast career is the Information Technology industry. Yaron has logged 15 years in software, 12 years as a DBA/Developer and has spent the last year as a DevOps engineer for Soluto (@SolutoEng on Twitter).
“You want to start with low hanging fruits to feel the wins quickly. This is a great way to find out more about the world you’re getting into.”
The biggest benefit is accountability for developers. Since we invested a lot of effort in empowering our developers to take full control of their code (from design to production), they also take on a lot of responsibility (answering pages, planning for scale, etc.). This creates high-quality code and products that are written in ways that allow them to develop and grow when the need arises. We see a lot of other benefits as well, such as integrating new technology quickly, improving most aspects of our development cycle continuously, and a higher production level for a number of processes.
A big challenge is not drifting back into silo thinking. DevOps in my company used to be a concept owned by no specific person. A decision was made to create a dedicated DevOps team when the company grew to a size that seemed to justify it (about 70 developers). The creation of the DevOps team accelerated existing processes and kickstarted a lot of high-effort projects that sat in the backlog. Unfortunately, it also created some “this is a Dev job and this is an Ops job” way of thinking, which requires subtle care to avoid falling back into silo culture.
Sit in a room and discuss pain points that DevOps can solve. Then try to think of the effort needed to solve each one and try to balance between profit and effort. You want to start with low hanging fruits to feel the wins quickly. This is a great way to find out more about the world you’re getting into. Avoid starting with really big assignments, as those will likely drag too long without proper experience.
Jason ford is a Lead Technical Ops Engineer at a full-service global marketing and ad agency. He has 7 years experience in Tech Ops 2.
“Also, since we have decreased the risk by using Blue/Green deployment strategies, we are able to do deployments during the day and not have to work late nights as frequently.”
The DevOps philosophy is integral to the continuing functioning and growth of my team. It has enabled us to automate, decrease risk, break silos, and improve release management. Before we started focusing on these philosophies, each deployment would be extremely manual, handcrafted, and with significant risk. Individuals would often have to perform “heroic” fixes and late hours on client environments.
We created an environment that would enable us to have consistent deployments that allowed us to balance work better, so that anyone who has some familiarity can provide coverage and allow people to have time off, and further breaking the silos of information. Breaking the silos is important because it reduces the risk of knowledge loss due to employee turnover.
Also, since we have decreased the risk by using blue/green deployment strategies, we are able to do deployments during the day and not have to work late nights as frequently. We trust developers to push code and are available if something goes wrong. This has increased our moral because it removed the gate keeper mentality and allows us to keep working on newer projects.
One piece of advice I would have is, while there is a learning curve, try to build everything as code. It’ll make your life easier.
Amy Boyd is the current CTO of CityMunch, an app that lets you find exclusive restaurant deals at your local favorites. Amy has 8 years experience working as a developer, and almost 4 years doing infrastructure work.
“Our AWS (Amazon Web Services) bill was cut by 30% when services started using auto-scaling Terraform modules.”
The biggest benefits for us have been high resilience, reduced server costs, and allowing us to make frequent changes with little worry.
Resilience is a high priority for CityMunch — we are an app that is used at payment time in restaurants and food trucks, so if our API was to have any downtime or serious bugs, it could frustrate customers and staff, and hold up an entire queue. Investing time in DevOps practices has helped us keep almost 100% uptime over the last year, while frequently deploying new changes, knowing that even a bad build will be rolled back automatically.
CityMunch currently operates in just one timezone, and sees peak usage during the day being 22x higher than at night. Having reusable Terraform modules and Docker practices allows different services to easily implement auto-scaling and instantly set up and tear down different test environments. Our AWS bill was cut by 30% when services started using auto-scaling Terraform modules.
The biggest challenge has been the pace of change. While the JavaScript community is much worse, I think the infrastructure automation world is also changing quite quickly. There are always new features in AWS and GCP (Google Cloud Platform) that can be taken advantage of; the tooling of Terraform and Docker with Puppet still feels immature and buggy; best practices seem to change every 6-12 months. It’s a good thing overall though – there needs to be a fast rate of change, because right now the tools available simply aren’t good enough for typical developers to want to integrate with operations.
Focus on what your users or customers want first. There is no point having perfectly automated infrastructure split across 3 continents if there isn’t a good business model and product. You will not go from 0 users to a million users in your first year, so delay operations work for as long as you possibly can.
Fabian is a Senior DevOps Engineer at Toast, with 7 years experience in the Information Technology industry. Toast is a point of sale system designed specifically for restaurants. Ponce also runs a DevOps consulting agency.
“Make sure your feedback loops are in place to learn from mistakes and perform root cause analysis on any significant problems and perform retrospectives on your sprints.”
At Toast, our rapid growth makes it difficult to ensure institutional knowledge diffuses its way into teams and all of our new-hire engineers. By being involved early in the design process for architecture and large initiatives, we can make recommendations and influence projects in a constructive way and educate developers about some of the value we can add.
One of the bigger challenges on our team is a direct result of how commonplace it is for other engineers to drop in and pick our brain on issues they’re having. This is a blessing and a curse, because while it takes away from our time for project work, being responsive to everyone in the organization is important in order to ensure our developers are empowered with the knowledge they need.
Don’t be afraid to be pragmatic. Best practices are hard to implement and can differ from company to company. Be upfront and communicative about the trade-offs being made and try to manage risk, rather than eliminate it. Align these risks with the business’ tolerance to them. Make sure your feedback loops are in place to learn from mistakes and perform root cause analysis on any significant problems and perform retrospectives on your sprints.
Switching to DevOps can seem like a stressful endeavor, but it doesn’t have to be. In fact, there are countless tools available to help you make the switch and stay on top of all of your DevOps needs! If you’re interested in learning more about DevOps and the benefits from real professionals, check out this post with the top 13 DevOps blogs available now.
With APM, server health metrics, and error log integration, improve your application performance with Stackify Retrace. Try your free two week trial today
Stackify's APM tools are used by thousands of .NET, Java, PHP, Node.js, Python, & Ruby developers all over the world.
Explore Retrace's product features to learn more.
If you would like to be a guest contributor to the Stackify blog please reach out to [email protected]
