Panel: WTF is Next for Developer Experience?

See why and how DevEx is essential in resourcing, recruitment, retention and innovation in our expert panel discussion with Suhail Patel, Hannah Paine, Crystal Hirschorn, Charles Humble and Jamie Dobson.

Key resources mentioned

How to Measure Anything: Finding the Intangibles in Business by Douglas W. Hubbard
The Varieties of Human Work by Dr Steven Shorrock

Panellists

Suhail Patel - Monzo

Hannah Paine - Container Solutions

Hannah Paine is an Engineering Manager at Container Solutions. Over the last 8 years she has found her groove building happy and successful engineering teams in consulting. She is a fierce advocate of individual and team happiness as the foremost key to successful client delivery, and has achieved success on large-scale digital transformations, across retail and government sectors, that demonstrates the power of people-focused leadership.

Crystal Hirschorn - Snyk

Crystal is an experienced VP/Director of Software Engineering with 20 years in the Technology industry. She has spent the majority of her career in the Media, Technology and Publishing sectors. 15 years hands-on software engineering experience; reaching the level of Principal Engineer before fully turning to management and leadership.

Crystal is an empathic and ambitious leader, striving to create the best conditions for teams to be successful, letting them take the credit for wins and providing a buffer to allow them to do great work. Passionate about diversity and inclusion in Tech with a track record of building diverse teams.

Hosts

Jamie Dobson - Container Solutions

Jamie Dobson is a Co-Founder and the CEO of Container Solutions, a professional services company that specialises in Cloud Native transformation. A veteran software engineer, he specialises in leadership and organisational strategy and is a frequent presenter at conferences.

Jamie also is the co-author of the book Cloud Native Transformation: Practical Patterns for Innovation, (O'Reilly Media, 2020).

Charles Humble - Container Solutions

Charles Humble is Editor-in-Chief at Container Solutions and former head of the editorial team at InfoQ.com. A former software engineer, architect and CTO, he has worked as a senior leader and executive of both technology and, more recently, content groups ranging from small start-ups to mid-sized and larger firms, working with off-shore, on-shore and remote teams.

Transcript Contents

Housekeeping and Announcements

What is the Goss?

Panel Discussion

Introductions 
WTF is DevEx? 
Can you give us a sense of the landscape of tooling that your DevEx teams are responsible for? 
When creating a high level of abstraction, how do you deal with the needs of the more infra-minded groups like data engineers? 
How do you gauge the effectiveness of the work you are doing? 
Do any of you have any experiences where the sort of developer perception and the reality on the ground are wildly different? 
How do you go about getting executive buy-in for having a DevEx function? 
Do you have any views on wider organisational issues? 
What are the pros and cons of having a dedicated function for developer experience versus embedding DevEx people in stream-aligned dev teams? 
How do you see the developer experience function evolving over time? 

Full transcript

Housekeeping and announcements

Jamie:
Okay. All of the people in the waiting room are now in the webinar. Hello. Welcome back to anybody who has been to our WTFs before, and welcome to any newbies who are here today. So whilst people are filing in, because it usually takes one or two minutes for people to actually get here, I've got a few announcements to make. Probably should start with some introductions. My name's Jamie Dobson. I am the founder, can you believe it, the co-founder of Container Solutions. We used to be a small company, and now we're less small. I am joined today by my colleague, Charles. Say hello, Charles.

Charles:
Hello.

Jamie:
Oh, very brief. Charles is going to be moderating today's panel, which is about what is or what the F is developer experience. How many people from the US do we have today, Carla?

Carla:
Well, almost half of the registrants, actually. So yeah.

Jamie:
You see, when we've got from the US, this dictates how much we can swear.

Carla:
No!

Jamie:
Because if we swear our normal amounts... Anyway, so I'll say it once. What the fuck? What the fuck is developer experience? I've done it now. Got it out of the way. And we are of course joined today by our wonderful panellists, Suhail, Crystal, and Hannah. Hello, guys and girls.

Hannah:
Hello.

Crystal:
Hello.

Jamie:
Welcome, welcome, welcome. I think this is your first experience with WTF. Fingers crossed and make the sign of the cross that it's not your last, that you enjoy this, and we meet again. Right. In the meantime, whilst we're getting ready, we have some housekeeping, some announcements, and then we have some gossip. Carla, who is joining us today with the gossip.

Carla:
Well. We have Rodrigo and Julian.

Jamie:
Rodrigo and Julian from Container Solutions will be bringing us the gossip. I hope it's good. I wonder if it's got anything to do with the falling price of sterling and the political catastrophe we find ourselves in this week in the UK. Housekeeping. We have a code of conduct. Carla will now post that into the Slack channel. So, in summary, our code of conduct is that we need to be respectful and mindful of each other and all of our divergent views. We do enforce the code of conduct. So any breaches in the code of conduct will be dealt with swiftly and definitively. If you are running events and you would like a code of conduct for your own events, feel free to steal ours. You can use it, you can edit it, or you can speak to me or Carla offline and we'll see if we could point you in the right direction.

Jamie:
If we are to maintain, to create and maintain safe spaces and the environment in the tech community that we're all responsible for building, we've got to take this stuff seriously. And you guys know that we do do that. What else in the housekeeping? Yes, questions please into the chat. Charles will moderate later when we get to the panel. But any questions or clarifications, get them straight in. We'll come back to you either in the chat or if it's part of the wider debate, we'll come into you there. As we go through the gossip and the announcements, we will share all links in the chat box. Please feel free to use them, tweet them, et cetera, et cetera. Carla, did I miss any big bits of housekeeping?

Carla:
Two things.

Jamie:
Two! It's been a long summer.

Carla:
Well, the first one is that, obviously, as you can see, this session is being recorded, and the video's going to be uploaded in our YouTube channel and the link is on the chat. Yes?

Jamie:
Yes. Yes. So if you've got a problem with that, then let us know now. You can switch your camera off if you don't want to be visible. If you forget to do that, let us know, and we can pixel you out. If you've got a real issue with being recorded, then you should probably leave the webinar and then watch it on playback.

Carla:
And the second one is that we have the live transcription on so you can also follow the subtitles live.

Jamie:
Now the live transcription is artificial intelligence, isn't it, Carla, that helps people who are hard of hearing or have deafness.

Carla:
Yep Exactly

Jamie:
The question is, can it handle my Yorkshire accent? Because Siri struggles tremendously. So this is a great test for the transcription service that we're using today. Okay. Thank you for that. Right. Onto the announcements then. So WTF is a newsletter, it's a bunch of webinars, it's a bunch of conferences. We are not a spammy company. So you can subscribe to the newsletter and you can consume all the content without ever getting a request for sales meetings or without ever getting asked if you want the job or stuff like that. If you are interested in that stuff, just click the boxes. We're dead proud of not spamming the people who join our database. So you feel free to do this, and we make it very easy to unsubscribe later.

Jamie:
But don't cry if you miss all the best content. The best thing that ever happened to me was somebody was really pissed off that they couldn't get access to the content, but they didn't want to subscribe to the newsletter. I was like, well, that's why we have this subscription service. And all of our stuff is free. All of our stuff is for free. There's no paywalls or anything like that. And a lot of it is driven by the community. So some of the contributions come from us, but a lot of it is brought by our friends in the industry, sometimes independent writers, et cetera, et cetera. Speaking of WTF, we will be running our SRE conference next year.

Jamie:
If any of you can remember back to the deep dark depths of the pandemic, we actually had a webinar with Nathan Harvey and it was called What The Fuck is SRE? And we got something like 700 signups for a webinar. This then gave us the idea, if something's working, don't change it. This gave us an idea to do a conference about 12 weeks later, which we did. So quite by accident, we invented a conference which is known as What The F is SRE? There's too many letters in there for me. WTF is SRE? We will be running this again in 2023. If you don't mind following that QR code or the link in the chat box, and let us know what you are interested in. We will tailor the content and the agenda to make sure that everybody gets something that they need. So if you fill in that questionnaire, we'll be able to put on a better conference for you.

Jamie:
Container Solutions is an awesome company. It's really awesome. Our management team consists of 40% female executives, 60% male. This is a number, somewhere between the 60/40, that we're trying to get throughout the whole organisation. It's diverse.
We're good at what we do. Everybody loves us. I think everybody loves us. I think that's a fair statement I can stand by. Oh, hello? A dissenter? A dissenter in the ranks. Somebody's like I don't love you. I just go for the free content. Anyway, it's a lovely company, and if you'd like to come and work with us, we've got all kinds of things cooking, engineering managers specifically at Amsterdam, even though we're actually a fully remote company, cloud native engineers. But we're also looking for two country managers. This is going to be dropped on the internet in the next couple of days. One to help lead our operations in Canada, another to lead things up in Germany. If you're interested in any of those roles, speak to me, speak to Carla, or have a poke around the careers page. And on that happy note... Oh, where's the gossip slide? We missed gossip slide.

Carla:
It should be there.

What is the goss?

Carla:
Rodrigo.

Jamie:
Rodrigo. Hello, Rodrigo. Welcome.

Rodrigo:
I'm the one. I hope it's good enough. So-

Jamie:
What gossip have you got for us?

Rodrigo:
Yeah. So, shocking. So Heroku is eliminating free access tier. For those who don't know Heroku is a subsidiary of Salesforce, and a Platform as a Service helps many people with DevOps, cloud engineers deploying service allowing people to build and run apps. And you can imagine everything like Go, Java, PHP, Scala. So after 10 years of offering free services, Heroku's pushing users to paid plans, and who is to blame? Well, quite often people abuse free compute tiers for fraud activities like mining cryptocurrencies and setting up phishing sites. And guess what? That what the case, unfortunately, fraud and abuse. And there was even cases where employees from companies faced several consequences because of things like mining bitcoin at work. And that sucks, right? So the company officially states in their blog, quote, "Our teams are expending an extraordinary amount of effort to manage fraud and abuse of the Heroku free product plans. In order to focus our resources on delivering mission critical capabilities for customers, we will be phasing out our free plan as well as deleting inactive accounts starting November 28th, 2022." That's really bad.

Jamie:
The tragedy of the commons. We have something to share and for us all to use, and people who are mining bitcoin fuck it up for everybody else. Okay. So they've been doing this for 10 years, free services. How many years after Salesforce acquired them have these services been for free? Is there any relationship between the Salesforce acquisition and this?

Rodrigo:
I think it has to do because now they have a bunch of competitors in the field. But the thing with abusing computers, it's really... Wow. I remind when he had a customer on previous jobs that the computer was being hijacked and they were exactly doing this mining bitcoins in their servers because of weak passwords. So...

Jamie:
If only we knew then what we know today, right? On that happy note, Julian... Thank you for that, Rodrigo. Julian, second piece of gossip. I like the Heroku gossip, by the way. I thought that was a good one.

Julian:
Sure. Hey. So, sorry it's not a goss, like last time, but I'm still going to talk about something that I think people should be aware of just in case they're interested. So today, the webinar is about DevEx. Something we don't think about usually about DevEx is the personal side of it. So what I'm thinking about here is the ergonomics of where you're working, but also how you work directly on the computer. So concerning the ergonomics, we usually think about chairs, so adjustable, chairs with adjustable pieces or standing desks. But something we usually don't think about is the keyboard. The keyboard is actually a really important piece of equipment, and before a few years ago, it was quite hard to get around a real ergonomic keyboard that would actually be useful and with a decent price. But nowadays you can have that kind of thing for very fair price. And I'm going to give you an example like this. You can really go check.

Julian:
It's the Keyboardio Atreus, and it has the shape of a split keyboard, but it's a unibody, so you have kind of an angle between your two hands. The idea here is to really have your hands be as straight as possible. And especially as devs, we are using a lot control keys for copy and pasting and other shift keys, and pinky, we use our pinkies all the time. And I don't know if anyone has ever had that kind of pain here along the way or on the other side. But this keyboard for example, have more keys around the thumbs. So like this, you avoid all those weird movements and you use your actual fingers that are on the right side of the force. So that's part of it. And going with this... Thanks to this, you can also have a really customised experience, because you can really program yourself, your keyboard, either with a GUI or with QMK, which is writing your own... That pain. I'm looking at the chat at the same time. So the pain is for everyone.

Julian:
So another part... The idea here is that you can customise it to your own needs, because we are all individuals and we don't have all the same needs. The other part of it that I just wanted to mention is that you can actually have the same ID for your development environment. And this is something that a term was coined, which is personalised development environment. And what is it? It's the same idea, that we are all individual and that we can actually have a development experience that is tailor-made to our own needs. So for this, I just wanted to mention that now there is something called Neovim, which is a very good piece of technology which is completely open-source. And if you think it's too much and you don't want to spend too much time on it, there is a fork of it called LunarVim, which is an opinionated, extensible and fast ID layer for Neovim. So all of this to say that you can also really personalise your experience to have the best time you want and be less tired at the end of the day.

Jamie:
Absolutely Awesome. Yes, your developer experience does extend to your physical environment. But what about those clicky keyboards that all you young people use nowadays?

Jamie:
It ruins my with my work experience...

Julian:
This is part of the custom of the keyboards that I were talking about.

Jamie:
Right.

Julian:
You can choose the shape of the keyboard, but you can also choose the shape of the keys and you can choose also how the keys react when you press on them. So you can have some... Depending on your fingers. Because we all have different fingers with different weight, and we can actually choose the right springs for you.

Jamie:
Very good. Okay, that was very, very interesting. I did not know that. So links to other developing environments in the chat box. Now, thank you to Julian and thank you to Rodrigo. It's at that moment I'm going to hand over to Charles. Round of applause for Charles and the panel. And we are going to start talking about what... Oh, oh, oh. Hello? What The Fuck is next for developer experience? Welcome to WTF.

Panel discussion

Introductions

Charles:
That's great. Thank you very much Jamie. So I do like to make these sessions interactive. We have had quite a few questions submitted already in advance, so I'll cover off as many of those as I can. But do keep questions coming in the chat, and again, we can put those to the panel as well. Maybe you could start off by giving us a little bit of what your are interested in terms of DevEx and maybe a bit of what your context is, because that really helps us just to frame the conversation. And while you are doing that, I will ask the three panellists to please introduce themselves. Suhail, maybe you could start us off please.

Suhail:
Yeah, absolutely. Thank you very much for the invitation to join, and thank you to everyone who's joined. Hi, my name is Suhail. I work on developer experience at a bank called Monzo. We are a UK-based bank with over 6 million customers. And for us developer experience is really, really important because if you think about traditional banking architecture and things like that, you've got to deploy systems really reliably and safely and securely, and typically that needs to really slow and really burdensome deployment processes.

Suhail:
Many folks who work in banks are very familiar with a six-month lead time in order to get any changes released into any sort of production environment. Change approval boards are aplenty, a lot of human review and things like that. And essentially, our goal is to eliminate all of that, automate that as much as possible. And we're now in a situation where we're deploying hundreds of times per day. And when I speak to my counterparts in other companies, they want to know the systemic approach in order to get to that world. And I'm afraid to say there is no silver bullet, but I imagine that will be a topic as part of the panel. Yeah, that's me.

Charles:
Great. Thank you very much. Hannah, could you maybe do your intro?

Hannah:
Hi. Good morning, good afternoon, good evening maybe to some. My name's Hannah. I am an engineering manager at Container Solutions. Previously, I lived life as a release manager, I've been delivery lead, all things in between, but I'm not an engineer nor have I ever been an engineer, which is quite an unusual position for an engineering manager. I think that's definitely brought a different aspect to what developer experience means to me and my teams, so I'm really excited to talk about that a bit more today.

Charles:
Fantastic. Thank you. Crystal.

Crystal:
Hello. Thank you for having me today. Also, I just wanted to say, if there's any cheering going on, the sales department's just out here. It's the last week of the quarter, so they're going a bit wild. I can hear them cheering, which it's a good thing. So I'm a director of engineering at Snyk. I run the infrastructure group for Snyk that's comprised of three teams today, which is cloud platforms, site reliability engineering, and developer experience. I actually got my engineering manager, Vlad, here on the call today, I've seen. So say hello. He's the DevEx engineering manager.

Crystal:
Why am I here? I have been a long time engineer, so I've been doing this for over 20 years now. Worked in all types of engineering, but most recently become way more interested in operations, platform engineering, infrastructure, and developer experience. I think it's... I've done a couple of these kind of things before on developer experience. I think it's such a broad topic, and it can be... It's such a vague thing, right? Developer experience, it can be anything and everything. So that how to bring a team like that into an organisation and make it successful is, for me, something that I really want to succeed at. And happy to share what we're doing here at Snyk with this team.

WTF is DevEx?

Charles:
That's fantastic. Thank you. So given that, as you say, it is a very broad term, I wonder if maybe I could ask each of you to give us a bit of a definition of what you consider to be DevEx, and maybe also cover off some of the related terms we use, so engineering productivity, developer enablement, and so on. And there was actually a question that came in from the registration form as well, which is the differences between developer experience and DevOps. So maybe we could touch on that a bit as well. Crystal, do you want to sort carry on with that one?

Crystal:
Sorry. Sure. I'm keeping myself on mute just because all the cheering. So yeah, when I went about setting up this team, this was the third team in my group, so it was the last one to form. They've been going for about, I don't know, probably about six months or so now. But it's like the thing that I always start with is the charter, right? That's really important to nail down. A lot of people might say it's a strange place to put DevEx, right, in an infra group, and I think, as I say to that team, it's a little bit arbitrary. It's just more or less I have the background of having teams like this, but also this is something I want us to set up for the company. And to be honest, in my mind I think it probably becomes its own department, its own group over time.

Crystal:
And that's often what happens at organisations that are more mature at this, say like GitHub, where you talk about things like engineering productivity. They have entire departments based around that, comprised of several teams. But that's not where we are. We're a startup. We're seven years old now, and this is our first foray into this. And I think for me, it's like it's really important to say okay, to the company, we have 300 engineers at Snyk today, maybe a bit more than that now. And it's trying to like quantify and qualify to them what is developer experience. Because, like I said, that's everything, right? Developer experience is everywhere, it's at every part of the stack. Somebody was talking before about the ergonomics, like how far does that go? And then we have things like operations as well. There's more things spinning up like product operations and how does that fit into the kind of holistic picture of things.

Crystal:
And so for me, I don't think... So I bandied about a couple of terms, and I thought about things like you said enablement, productivity. But I think it has elements of all of those things, and I felt like calling it productivity or enablement was probably not the angle that I wanted to go for. I want it to be called developer experience, because it's like how do we bring that kind of joy, and kind of day to day, how do we bring quality of life improvements to engineers and move that needle forward for them on a continuous basis? And so it just depends on what that is as well and how to fix that.

Crystal:
So it requires a whole systems thinking approach. Because sometimes we might focus on the CI pipeline and how fast that is. And this is something we did at Snyk... Which I can go into more around DORA metrics and the way that we're tracking that, but one of the things we uncovered during that process was actually the PR life cycle was the thing that was causing the tension. It's getting people to put eyes on PRs and get that review done. That was actually the thing that had the lead time. So it's quite trying to also uncover non-obvious areas of friction as well and reduce that over time. I'll stop there.

Charles:
Suhail, do you want to come in on that?

Suhail:
Yeah, absolutely. I very much agree with Crystal's definition, and that's the same approach that we go for at Monzo. For us, developer experience is everything from the ergonomics of being an engineer, the tools that you use, the editors and things like that. For us, it's very important that we standardise for the purposes of security and audit and compliance and things like that, but we give enough freedom for engineers to go out and experiment. If you ask a lot of engineers in many traditional companies and companies that I've worked in, even in the past at startups, every engineer has their folder of shortcuts, folder of scripts or whatever, things to make life easier for them. And typically those get shared around by word of mouth or hidden somewhere in some sort of GitHub gist or maybe a Slack attachment or something like that. How do we bring those and make them mature tools that an entire organization can adopt?

Suhail:
Developer experience can go even beyond just engineers as well. For example, there's questions that engineer managers want to answer and engineering leaders want to answer and things like that. How can we empower them to also feed into the picture without spending a whole heap of time setting up their entire environment and essentially doing all of the effort to become an engineer. For us, I think this is also quite an early function, even though we've been going for a very similar amount of time to Snyk. It's quite an early function. We've accelerated quite far into standardising around deployment and optimising CI pipelines and things like that.

Suhail:
But I think it goes beyond that as well. So, for example, how do you work in your IDE? If we've standardised around particular languages and things like that, how can we make that sort of joy of programming? So, for example, I don't know if folks have played with GitHub Copilot, which is like an auto suggesting editor experience. A lot of folks installed that and we're like, this is magic. It is auto-completing my code based on context that I've put into the editor. How can we bring that joy to engineers at our scale, at a scale that is relevant for us? We want to bring that spark back for engineers.

Suhail:
I do want to touch on one of the points that you raised earlier about the difference between DevOps and developer experience. DevOps is essentially bringing that joy into operations. So if I take a view back into the past, if you want to do things in an operational manner, if you go really, really far back, folks were hacking around with Bash scripts and trying to write their own bits of custom automation and things like that. SSHing into servers and things like that. And the DevOps movement has made that significantly more automated, significantly more iterative, things that you can share, things that you can depend on. And there's been a whole heap of tooling around this whole infrastructural space. So it is almost like bringing the joy of developer experience into the world of operations. So I essentially see it as a subset of the world of operations.

Charles:
That's great. Thank you. Hannah.

Hannah:
Yeah. It's nice that we're all in agreement. It makes it quite a nice environment. I think, as Crystal said, the holistic aspect is about improving quality of life, not just your technical systems, tools, and processes. To me, DevEx is how we make the what we're doing easier, a more enjoyable experience for engineers. It doesn't really matter what the what is. It could be a product, it could be a project, it is whatever is the focus. It takes cognitive, emotional, and capacity pressures off all of the downstream engineers. I think that's the goal. And it's not just about enablement. It's about enjoyment too, as we've all mentioned. It's about enjoying what you are doing, and, as Julian mentioned, like right the way down to your physical setup. And does the sound your keyboard make bring you joy? Or does it bring you anger? In which case, needs addressing. I think from the DevOps to DevEx comparison, to me DevOps is a function and set of processes focused on improving the end-to-end delivery of engineering work. DevEx to me steps a level and looks at it more broadly as to the human element of DevOps.

Can you give us a sense of the landscape of tooling that your DevEx teams are responsible for?

Charles:
Yeah. I think that's great, actually. Thank you. There's a lot of comments in the chat about the idea of introducing joy into engineering work, so that seems to be really resonating with people. Can you give us a sense of the landscape of tooling that your DevEx teams are responsible for? Suhail, maybe you could talk a bit to that.

Suhail:
Yeah, absolutely. So for us, pretty common bits that are included in this sort of team are things around continuous improvement and continuous delivery, testing pipelines, things around deployments, and things like that. But actually, where I think the landscape is, is going towards, is more towards observability around your systems and services. So not talking about like metrics and logging, not runtime observability as your systems are running, but observability like how do your systems live? Where does the documentation live? How do you do service discovery? How do you determine that an endpoint exists and whether it's internal or external? How you should call it, how you should authorize against it, and things like that, things that would typically be loose documentation, making that a lot more structured. You see that a lot with systems like Backstage, for example, which is like a service discovery style of Tool. And I see it creeping more and more into other spaces like machine learning and things like that. What kind of models exist? What were they trained on? And similar things.

Suhail:
I've mentioned documentation, bringing structured documentation, both generated documentation from the API specs but also bringing some sort of consistency around documentation tooling to encourage better documentation and more enforcing some sort of structure, especially as organisational complexity grows. And yeah, it's like a conglomerate of all of these different functions. I hate to sort of describe it as like the other team. It's not just a grab-bag of things. All of these are focused around making the organisational complexity lower for engineers within a large organisation. So I think a natural question is at what size do you start developing such a function? Once you start establishing that all of these different areas are trying to solve the same problem, it does make sense to start spinning up this sort of function and standardise on some of this tooling so that you can benefit from all of these optimizations from an organisational perspective.

Charles:
Crystal, maybe, do you want to come in on that?

Crystal:
Yeah, sure. I just want to expand on one of those last points that you said, Suhail, around... I think it's like a lot of the companies don't do this until they get to a certain size and maturity. And so I do think kind of sub-50 engineers, I don't really think we need a dedicated DevEx function here. Kind of when you get to around the 100 mark, that's when processes and tech and kind of the principles that go along with that, in terms of keeping the bar raised in terms of even the way those things are codified into your tools and your platforms, that's when it starts to break down, I think, quite a bit. And I think one point that Hannah made there too, it's like it's not just about engineering. It's also about products in a lot of ways too. And if you have program and project management, how do these functions cross-functionally work together at a bigger point and scale and as you go through those kind of inflection points as an organization.

Crystal:
So that's like why, for us, it was the right time to do it. I wish it probably had been done sooner, but it's where we have started to see breakdowns in that. In terms of things that the DevEx team does own at the minute, though, it's quite similar. Anything that goes around the CI/CD pipeline is within their remit. So things that we use Circle CI or GoCD is kind of the key thing there. So you can, from that, infer that we're kind of largely using Kubernetes at Snyk as well and a lot of Helm. And one of the things that's on the radar for DevEx, this coming quarter in fact, is we have built an internal runtime platform for engineers on top of the cloud providers, and we're trying to create a higher level API for engineers. Because right now the Helm charts and the YAML files that set in the repos, it's like, well, now I need to know all this stuff about infrastructure.

Crystal:
And it's like, no, no, no, no, no. With 300 engineers, they shouldn't really have to know that much about anything, right? And if we get to the best place we could be, which is the Heroku thing, which is like I just want to spin up a service. I don't really give a shit. I just want good defaults. I want you to tell me what the best practice is. Just do the thing for me. That's like the 90% case, or maybe even 95% case. And so that's where we're aiming towards, is like create the higher level API and then get the service templates spun off the back of it. And then that's where you can talk about things like Backstage and it being a key piece of the technology platforms that we're trying to drive at Snyk. And why this project from Spotify is so interesting and so important as well is that we can drive things like that from it.

Crystal:
It's like okay, we have a good API. Let's just get the services spun up. If you have a specific need, there are extensibility points there too so that you can kind of say, well, I need a specific database resource and I need it to be configured like this. So the engineers still have that ability to kind of have some control over what they're doing, where they feel comfortable. I think that that's the key point as well, where they feel comfortable. And that's like a whole set of maturity steps that organisations have to go through. It's just a thing, right? And so that's what we're going through now. And, yeah, I guess... What else would I say?

Crystal:
I'm trying to think what else they own. There's a lot of things, but Backstage is one of the kind of key ones that we're looking at this coming quarter. And that's more bringing vendor tech. So it's not like a "not invented here". We try to be very careful of that. So it's like let's use something that's already working and make that successful. And even bringing that to bear, things like Scorecards or Tech Insights, as they're called. So how can we drive some amount of also accountability into the organisation and give them the data into the hands of every single team to. So Scorecards are amazing. So if you don't know much about that, I'd say look at Backstage, look at Scorecards. This is kind of our big focus for the next couple quarters at Snyk. I'll stop there. There's probably other stuff that I've completely missed that they own and they do, but that's enough.

When creating a high level of abstraction, how do you deal with the needs of the more infra-minded groups like data engineer?

Charles:
No, that's great. Thank you. There's a sort of a related question that's come in from Pavel Brodskym apologies if I'm mispronouncing your name there, which is when creating a high level of abstraction, how do you deal with the needs of the more infra-minded groups like data engineers, et cetera, that need to have control that's outside of the sort of 90 to 95% happy path?

Crystal:
Yeah, sure. I mean I could quickly answer that. So when we built this platform, we kind of looked at it from a point of view like there are layers to it. So we came up with five layers, and of course, the kind of ground layer zero is just how do you get an account spun up and kind of the basics around an account and the org structure within the cloud provider itself.

Charles:
Mm-hmm.

Crystal:
And we'll eventually add things like GuardDuty and stuff like that to that process. And then you have things like networking all the way up to the application there, right? And so at each point of those layers, there are extension points. And so one of the things that I say... We had an engineer recently who needed a MongoDB resource but cloud platforms hadn't yet created that resource for our platform. But this person came from an infra background, so we're like, do you want to write some Terraform? And we're like here's our playbook, and they went away and did it in like two days and then they told the rest of R&D at a demo. Like this is something we did. Hooray. It's super easy and super quick if you're comfortable enough doing that. But of course we wouldn't expect engineers to do that.

Crystal:
So I think one of the things that we haven't touched on is capacity. You can build a lot of stuff, but for a DevEx team, in terms of the way it should look at its capacity, there's a lot that goes into education, pairing, consultancy, documentation writing. And so we try to make sure that there is an amount of capacity always set aside for activities like that. Actually across my whole group. But that's one way we manage that, is if you're not comfortable, this is the way that we can help you and get you onto the paved road as well on our platform.

Charles:
That's great. Thank you. Hannah, do you have anything to sort of add in on this?

Hannah:
I really like the idea of the tiers, because one of the biggest challenges I've seen working either in a massive company versus a company going through scale up, is the perception of what is an infringement on autonomy versus what is an enabler and takes cognitive load off? And culture has a huge impact on that as well, so what your culture of independence is. I think from a DevEx perspective and infrastructure perspective, it's just not seen as sexy enough. I don't understand why, because I think it's great. But in my experience, often the sexy work is seen as that application development. So taking the load off from having to think about the how do we get to even having an application deployed generally does work out okay when we're talking about restrictions and autonomy. Standardizing of that, teaching them how... This is where I think the DevOps feeds in nicely. Teaching kind of software and application-level developers how to enable themselves to the right degree allows you a mixture of the standardisation with empowerment and autonomy.

Charles:
Thank you. Suhail?

Suhail:
Yeah. I completely agree with everything that Crystal and Hannah have mentioned. It doesn't need to be restrictive, but I think there's a slight aside to that which is a lot of the times, a lot of these teams sort of focus on the 5% edge case rather than optimising for the 95%. And you sort of end up in this situation where nobody is winning. I think it's important to optimise for the 95% case, which is, and as I mentioned in the chat, many engineers do not care about running infrastructure. They want to stay on the paved road. So if you make the paved road as swift as possible and allow extension points to add to the paved road where necessary... A developer experience team doesn't sort of work in a bubble. Just like as Hannah mentioned about culture, this is where culture really comes into play, where folks feel like they have the autonomy to contribute to the developer experience because they're using it on the ground on a daily basis.

Suhail:
They are the people... One of the things that I see is that the team that is working on developer experience, they can do as much user research as in the field, right, but they don't have a true feeling about what other engineers are feeling on the ground, the frustrations that they are feeling on the ground, especially as they are developing and deploying applications. So getting a good understanding and allowing other engineers to contribute and giving them that guidance and consultancy, as Crystal mentioned, in order to add their own extension points where necessary and extend on the paved road, it is a global effort. I see the DevEx team as just mostly maintaining this global effort. So for them, it's all about making sure that they're adding things that they can continue to support and things like that. Having a centralised team to go to, but they are not the be all and end all. They shouldn't be gatekeepers. They should be enablers.

Hannah:
Yeah. To add to that, gatekeepers not enablers is absolutely on the line, exactly that. I think there is also the element of how do you enable a DevEx team to be successful? And where I've seen it work really well is where it's almost treated like a product who has end-users, who you had to have a product manager who goes out, gathers requirements, understands how to prioritise, works with your technical leads to understand where have we got a balance of quick wins versus actually this is a bigger piece, and the communication then.

Hannah:
All of the responsibilities as if you were going to johnlewis.com and buying a product and it was your responsibility as the product owner of checkout to be understanding, well, how long does it take for my people to check out and how can I improve that? What do we need to do behind the scenes? What can we do up front? How do we communicate those changes? How do we trial them? What do we measure success as? And that sets up then your DevEx team for success, because there is also the balance of like they're engineers as well. How do you go to make a good DevEx for your DevEx team when their priority is making it a good experience for people downstream.

How do you gauge the effectiveness of the work you are doing?

Charles:
I think that's really interesting. We have a lot of questions around how DevEx teams validate the work that they're doing. So how you measure, what you measure, how you gauge the effectiveness of the work you are doing. I don't know. Hannah, maybe you could start off on that?

Hannah:
I think that's a really challenging one from a consultancy perspective as well. And I think [inaudible 00:41:38] also asked a really good question kind of related to this around balancing how do we make this a wonderful experience for our engineers versus but what do our users need, as in down the line? What do our clients want and need? What boundaries are we working within? And this is where I find it super important to look at the non-tooling priorities within DevEx.

Hannah:
Because I don't have necessarily control of what my client uses. I don't have control of how their pipelines necessarily look. But what we can influence are the other aspects within the developer's experience that can be a positive environment. This is our internal processes as a consultancy. It's the psychological safety you provide. It's your physical setup at home. But, yeah, I mean we're talking a lot about joy. Being pragmatic, it doesn't always bring joy. So how can you lessen the burden in the areas within your control? And in consulting that's a really, really challenging area, and you just have to hope that you also have some positive cultural influence within your client. But not always.

Charles:
All right, Crystal, you mentioned sort of DORA metrics earlier, quite a lot earlier on in the conversation. Is that sort of where you started off in terms of thinking about measuring?

Crystal:
That was actually the very first project the team did as a team when they were coming together. They kind of were split off from cloud platforms originally, so that's where it kind of incubated and then got enough people in the DORA to kind of create that team. So I thought it was a fun project, talking of fun, a fun thing to do, kind of more experimental. Kind of kick off the team in the right way and the kind of mindset that they should take on for the long term. And there are platforms out there that we looked at, and I'm not going to throw too much shade at them. But we looked at things like Velocity, Code Climate Velocity, and the thing that I found with Velocity was I wanted to give it a try, had kind of looked at it in previous organisations, but I just found it really overwhelming. And I was going around different groups in Snyk trying to explain what it might bring, and the amount of questions, it just...

Crystal:
Like sometimes some groups were asking me more than a hundred questions about what they were looking at, and I thought this can't be good. But it also... The other thing that you have to, when you're talking about measurement... Which one of the things that put me off that platform a little bit was it seemed to suggest that engineering managers could then use it for ill in, like terms of performance rating. How many commits is this person making per day? Or how many PRs? I was like no, no, no, no. So you have to be really careful that isn't the culture you're bringing into the organisation because that kills it immediately. So for us it was like it's not personal, and anyway the unit of measurement is the team. It's not the individual. So that should always be the case. But we wanted to look at it even higher level. Going back to something Suhail said earlier, was for me it was also about enabling engineering leaders and product leaders to make better decisions and trade-offs when we come into quarterly planning.

Crystal:
People talk about tech debt, and that's very nebulous and kind of not very quantifiable. It's like saying here are some KPIs, here's some measures that can tell us where things are actively going quite wrong today. And one of the things that uncovered... It was like we had a gut feel about things as well, and our CI pipeline... To explain Snyk has a monolithic architecture, and we're breaking that down, as companies likely do, and moving to service-oriented architecture. But we already had this feeling that our pipeline was just... it's just around that monoliths. It's terrible. But putting some actual numbers against it was interesting. And also quantifying what is a failure case versus a success case was also interesting. So like we were saying okay, if from our definition of failure, it's failing 80% of the time on the first build. And then we broke down why, the reasons why, in a graph.

Crystal:
So anybody could go and look at that, but also we could say, okay, what does this look like for this one monolith versus any other services that set outside of it and do lots of side-by-side comparison. And that was really useful because that kind of said, okay, well, there is a lot of value in spending the investment across R&D to break down the monolith, and this is the kind of return on investment that we shall see. So this is the thing. How do we as engineers talk in business speak, basically? How can we quantify that and advocate for the work that we want to do? So it's like being able to put some measures against that was really important. And we kind of look at that pretty consistently.

Crystal:
I think one of the things that kind of struck me a little bit is when you read the Accelerate book that talks about DORA, they're looking at such a large swath of industries that I was saying, oh, we should put the quartiles on there, see where we're comparing against industry. Because I was thinking, oh, we're not tracking that well. But if you push code to production more than once a day, you're already in like the top... And I was like, okay, maybe we need to move this into where we think we should be given that we're a Cloud Native young company and we're making 60 to 70 pushes to production a day. What should that number be versus what is the book telling us to do? And that was an interesting takeaway, but only happened after we did most of the project.

Charles:
Yep. Suhail, in terms of measuring the impact of what you're doing, how do you go about that at Monzo?

Suhail:
I think there've been... It's very refreshing to hear both Hannah and Crystal's take because we've tried many times to be very objective about how we measure the impact of developer experience, and I'm afraid to say we've always failed miserably. And you come to a very natural conclusion that it is quite a subjective measure. For example, if you speak to a lot of engineers, especially folks who've worked at Google, they will rave about the internal Google tooling, and similar for companies like Facebook. But when you ask them to pinpoint any one particular thing, it's just everything. Everything. They talk about everything behind the scenes. So, for us, it's very much based on gut feeling, as Crystal mentioned. Gut feeling and things that we want to empower, right?

Suhail:
So I think the wrong way to look at it is how do we make our engineers write more code? I think it's the wrong question to answer. I think it's better to say, okay, where are our engineers actually finding it difficult to write code? Or for example, why are my engineers not able to deploy on a Friday? Or why are my engineers not able to deploy at 5:00 PM? Or something like that. Where does that psychological barrier come from? Are they worried that they're going to break the environment or leave it in a state where they can't leave it behind or cause problems for the on-caller or things like that? These are psychological barriers. And focusing and zooming in on those... And those are based on gut feel. Like that takes research, that takes time. I think we, as engineers especially, we want to take the quick and optimised path and say okay, right, we're going to put a number to this, and the thing that is worst, we're going to focus on that.

Suhail:
And there've been many valiant attempts, and a lot of them have ended up in miserable failure. Now, gut feeling also can end up in failure, and I can give you plenty of examples where there has been the case for us. But I think it's been good that we've tried and dove deep into the problem and sort spoken to a lot of engineers and taking that time to do that user research, treating it like a product that we release into the wild. You need to do that user research. You can't just look at a score. There's a whole reason why user research function exists for products, and I think the same needs to happen for effective developer experience tooling and teams.

Do any of you have any experiences where the sort of developer perception and the reality on the ground are wildly different?

Charles:
Do any of you have any experiences where the sort of developer perception and the reality on the ground are wildly different?

Suhail:
I could maybe go.

Charles:
Oh, okay. Good.

Suhail:
I could maybe go first on that one. Absolutely. I think the speed, the speed at which something happens is typically one of the biggest gripes. A lot of times, something is slow, right, and slow is a very relative measure. Slow in comparison to what? Slow in comparison to prior experiences or slow in comparison to how fast you feel that this should be happening or slow relative to the speed that you were expecting for a particular thing to run, which makes you drastically unproductive as a result? There's a classic XKCD meme which is my code is compiling, right? And code, if you use a compiled language, your code needs to compile. And then there's relative measures of slow. So for example, if you're working on a project and that project is a very complicated project but you're working on it for a very short period of time, you will likely optimise the slow compiled times in order to not have to reinvent the Earth.

Suhail:
But if you're working on a product on a day-to-day basis and releasing it many, many times per day, you will want to optimise, for example, the compiling experience or the deployment experience and things like that to make it as quick as possible. Either throw money at the problem or throw engineering effort at the problem. And a lot of engineers don't understand that for developer experience, it's a very hard team to justify, because they're not shipping anything that is objectively towards company goals, right? They are one step removed.

Suhail:
A lot of us talk about, for example, undifferentiated heavy lifting and things like that. If you look at a company like Amazon or AWS or any of these different providers that provide a SaaS, their product and their goals is part of their product. They are building this product, they are selling it to you. For them to make it fast and add new features is part of the product offering, right? Whereas, when you are a team removed, when you're an internal team, when you're building it yourself, you are one step removed. And there is always an opportunity cost, because those engineers could be working on a product and it's difficult to justify head count because you need to justify how much more is this going to empower my engineers. And it's a really, really tough question to answer.

Hannah:
Can I add on top of that from a consultancy perspective? When you are then trying to convince a client to spend more money on an intangible thing that you know will improve, obviously improve your engineering experience, but it is going to improve speed of development, it's going to improve engagement and the intangibles, you're right, it becomes harder and harder as you become further removed from the direct metrics of this will turn over this amount of money and add this amount of value to what we are doing.

Jamie:
I think there's a difference between metrics you can measure in real time, such as number of check-ins, check-ins per day, then hard metrics such as meantime to failure, meantime to recover. But the DORA stuff is also around how much quickly you can turn experiments around, and that in itself leads to a culture of psychological safety. Those will only be measured with trailing indicators, retention of your staff, employee engagement service, cash flows, profitabilities. So there's an issue. So, Suhail, you mentioned that you've tried to measure this objectively. The hardest things are hard to measure and often can only be done in retrospect. And I think we should not kid ourselves as engineers... We're all engineers, right, so it's all about how many check-ins do we do? But you need to take a step back and say, well, what's that DevEx contributing to the wider metrics around safety, experimentation, and just satisfaction in your job. And we often forget that because we're so busy doing what we're doing on a day to day.

Charles:
Yeah, I think that's spot on, Jamie. Thank you. Crystal, I saw you nodding to the sort of difference between perception and reality question that kicked this bit off. Did you want to come in on that?

Crystal:
Yeah. I mean, I think that happens in a lot of cases is the truth. One of the things that I haven't heard yet is the kind of broken windows scenario, where we become adjusted to things that are just slow or broken over time and desensitised to them almost. Whereas, newer engineers will come in and be like, what the hell? But we'll be like this is the norm and this is what we've come to expect. Like the thing I was talking about, the monolith.

Crystal:
It bugs me a lot because when I joined Snyk, it's like we had a specific, what's the word, working group set up, and I threw a person in it as well to kind of say, okay, let's get this build time under 20 minutes. And they're like, okay, we can do it. And we got it, we did, we got it down to under 20 minutes. It's like 18 minutes. And now here I am two years later and we're saying the build time, we want to get it to under an hour. And I'm like oh. So that's the kind of thing with growth, right? These kind of things become normal, but they also kind of get exacerbated I think a little bit.

Crystal:
But I think that's why the DORA metrics project was so important to me, is because I could show them. And we were a little bit worried actually when we did that project because we were like, well, from our point of view these things are failure cases. All of these things, flaky tests, failures, that's a failure. We would say this all fails. If the artefact isn't built and ready to be pushed up into some environment, it's failed. And so that's where we kind of... Okay, 75%, 80% of the time, it's failure. And we were worried to show that to engineers because they might have a really bad reaction to that and might start pushing back. But that's why we tried to be very careful about how we labelled things and sort of said this is the breakdown. It's not just some red/green percentage, but here's the breakdown of what it is. And that also helps management too, in terms of identifying particular areas where more investment is needed.

Crystal:
And the last thing that I'd mention here is building empathy is really important. So one of the things we're going to do with developer experience team at Snyk this quarter is making sure that every engineer on that team spends at least one sprint in a team that's not their team to try and see... Because there's this thing called work as imagined versus the work as done. And look it up. There's something called Varieties of Human Work, and that's what the article's called on the internet.

Crystal:
And again, changed my whole way of thinking about this. But it's like we need to know what it's being on the sharp end and developing that stuff, like from an application engineer's perspective. What do they need? Let stop assuming what they need and figure out what they really need and where the highest points of friction are. Because that's what I meant about like quality of life. What is going to move the needle? We can't just assume that. We need to go and live that life with them and figure that out together.

How do you go about getting executive buy-in for having a DevEx function?

Charles:
Yeah. Hannah, you started to talk a little bit about the business of getting sort of of buy-in to invest in developer experience, in a sort of general term. I was curious if you or anyone else wanted to talk a bit about the whole business of getting particularly kind of executive-level buy-in for having a DevEx function for putting time and energy and investment into it.

Hannah:
I can talk around where it's been a struggle.

Charles:
Yep. I could do that too, but go on.

Hannah:
I think,... And I'm going to harp on a little about this, but the difference often comes between product-led businesses and consultancies. So I've spent my career working in consultancies, and invariably, we will get the buy-in down the line when things have started to go wrong and we say this will really help fix something that's already wrong, rather than this will improve and make things better in a way that you can't see straight away. I don't think I have a solution for how we overcome that within consultancy other than starting to also look more at, well, look, the wider industry and successful product businesses do invest in this. There is evidence already as we take it to our clients and we're also talking about improving their culture around engineering and delivery. That, unfortunately, from a consulting perspective, is a catch up game. Maybe I'm a cynic, but it's very hard to be a leader in this kind of investment in intangible improvements as a consultancy.

Hannah:
We are often looking to product companies that have done it successfully and say, to our clients, look, if this is who you want to be like, this is where the direction you want to travel in, look how well it is already working. Please allow us to also do this for you. So yeah. I think this is why I like... And I've also reiterated around culture is such a huge, huge one. As a consultancy, we are absolutely invested in developer experience. We understand the impact it has without needing to go through the process of convincing senior stakeholders who hold the purse strings that that is the right thing to do. I think if you can set up a culture from the beginning within that company, you are already onto the right kind of path, and then it becomes a case of ticking the right boxes within your finances as well to say... and using those metrics to help fuel that says, look, this is the successful thing to do because we can see it's successful without having to go through the pain of it being really bad first.

Charles:
Right. Yes. Crystal, Suhail, anything to add in terms of exec buy-in specifically, maybe?

Suhail:
I think there is one key indicator that is pretty important, and unfortunately, it comes during when engineers leave. When engineers leave, you get this magical wave where they open up and talk about many of their frustrations. And one of the things that comes up pretty often, especially when you've not invested in this particular area, is the poor use of tools. The frustrations around getting things shipped is a very common reason why engineers will choose to leave their particular area. So if you look at a lot of the companies that have been successful, there is a correlation in their success where they've invested in engineering tooling in order to allow them to ship quicker, reduce the meantime to failure. All of this is part of the developer experience, right, giving... Again, removing that psychological barrier to shipping stuff.

Suhail:
And as Crystal mentioned, the kind of everything that... like a flaky test, a bug that's crept into the code, a deployment that's been borked, right, all of these are failures, right? These are all paper cuts that engineers will experience. And when engineers become accustomed to these broken windows, they will be less productive as a result. Because there'll be a class of engineers who will be very blasé and just fling stuff over the fence, and there'll be a very different class of engineers who will want to be very, very careful. And that will ultimately slow them down. These are all trailing indicators, as Jamie mentioned. And unfortunately, you find out a little bit too late. But I don't think it's irreparable, even in every size of organisation.

Suhail:
You don't get to a large organisation, say, okay, all hell has broken loose and there's nothing we can do about it. We just have to survive. There are things that you can start, and I think the case is easier to make even at a larger organisation where you have potentially some headcount available. You can say this is an area where a lot of engineers are experiencing paper cuts. Here is some data. Right? It takes a bit of gut feeding to identify. Here's some data to prove that this is a problem, and I think with these we can get this graph down. And say okay, right, with two engineers' time worth of effort, we can make 100 engineers worth of productivity. It's a fairly easy equation to sell.

Hannah:
I think metrics and trust.

Suhail:
Trust, yeah.

Hannah:
Comes down to trust that the experiments will be worth it and trust that if they're not, you continue to learn lessons from it.

Charles:
Yes, I think I'll add to that as well, if you can have metrics that relate in some way to things that the executives care about. So your average executive is not going to care about, I don't know, number of times your application crashes or the number of outages there are or something like that. But if you can relate it to lost in a process or something that the exec team do actually care about, then it then becomes much easier to back sell that, I think. Sorry, Crystal, you were about to come in with something there.

Crystal:
Yeah. I think it's actually... To your point there, Charles, so a couple... I mean, we had the buy-in, so I was lucky, I should say, because I've been in organisations where that definitely wasn't the case. So that's refreshing from my point of view, that there was trust. My SVP was like, this sounds like a good team to have, and I need... He's like, when I go to my exec meetings he said it can be a bit frustrating because they'll have reports, a lot of them will have reports and data they can share, but he said I as the engineering leader often don't have a lot of things to base that on. So we talk about things like customer SLAs and uptime, and he said, but we all know that these are measures that are just not that good. And he said, so I would like to have more data as well, in terms of how is the kind of heartbeat of engineering and how can we make that more productive?

Crystal:
And so that was good in terms of a buy-in. But one of the things that we kind of did was... So I posted a link to a book in there which is How to Measure Anything: Finding the Intangibles in Business. And I would say go and read that book. It's amazing. It can be a little bit dry at times. It does talk about a lot of statistics terminology, but not too deep, so anybody without a statistics background can get a good handle on it. But it's kind of, again, another thing that's completely changed my way of thinking. And so we came up with a cost of delay model at Snyk, which is like, okay, here is a spreadsheet... And we're not totally fixated on build time, but that just happens to be where we started. But we have said okay, how long is the build? How many engineers do we have? How much do we pay them? And if we brought that down by X percent, how much money is that going to save the business?

Crystal:
So this is I think big, because I used to be a VP of engineering, and I think having been in my boss's chair before, it's like I remember what that was like, how to talk business-speak, like how do we justify ourselves? And I think engineers often find that tricky, in terms of talking the ROI and the business-speak that needs to happen. That's where the buy-in comes from, is being able to quantify things and to talk in that kind of business-speak way. And so we were able to say, well, yeah, if we make this investment we can save 500K a year pretty easily. Which it doesn't sound like a lot, but at our scale then, which was less than 100 engineers, that was pretty significant.

Crystal:
Now, it's way more, it's probably a couple million a year. And you think that's just by shaving 10, 15 minutes off of a build time. And so surely that justifies the team just by itself. It's like you almost have to do nothing else. But there's a lot more value that we bring, right? And so one of the... We're saying okay that's a good place. A lot of people understood that. How could we take that same model and apply it into SRE? So what is the cost of an incident? And, again, sure it's going to be ballparks and that'll get more and more quantifiable over time, but let's start with some sort of model that gives us a range that we can then talk to the business about. Like why do we invest in learning from incidents? It's because we expect this kind of ROI from it. And I think that that's something I would say, a good thing to think about to take forward into any organisation in terms of getting buy-in.

Do you have any views on wider organisational issues?

Charles:
I was curious as to whether any of you have any thoughts on sort of wider organisational issues? So something from my own experience is we tend to focus a lot on getting our CI/CD pipelines really slick or being able to deploy it super frequently to production. But perhaps that's not our biggest issue. Perhaps our biggest issue is actually the requirements we're getting just aren't very good or aren't very clear, or maybe there's a problem in the product management process or something like that. So it's not really within the direct purview of DevEx, but as we've been saying, DevEx is kind of a wider thing. I don't know if anyone has any sort of thoughts on that.

Suhail:
One of the things that I focus on and I care about very deeply is the creeping in of organisational complexity, like when it comes to... with respect to develop efficiency. So, for example, very concretely, in many teams, if you want a system to be changed that your team doesn't directly own, typically you'd go and add it to the other team's roadmap and beg probably their EM or lead to prioritise your thing so that you can ship your feature. I'm sure that's going to be a very common... That's going to resonate very widely across this particular group, when you're relying on someone else. And with a lot of the advent of things like monorepos and things like that, that kind of stuff has become significantly easier, where you can go in and help contribute changes to other areas of the business. And things aren't sort hidden away and the complexity is sort of self-contained in one particular area.

Suhail:
But some of that still can be barriers to entry. For example, if a different team is using a different programming paradigm or using a different programming language and things like that, you're still going to have that organisational barrier where you don't have all of the context. So, for example, we were talking earlier about the paved road, and the paved road can also include things like the programming language that you use or how you structure your applications, where your code lives, how you define your HTTP handlers, how you communicate between services and things like that. Anything to make cross-company collaboration significantly easier can drastically reduce the meantime to delivery in general.

Suhail:
Because instead of focusing one particular area in one particular team, you've got a whole swath of engineers who are willing and able to contribute across different team initiatives. Now, that comes with its own potential problems around ownership and every engineer wanting every feature under the sun encapsulated within systems and services. And that's a side that you need to manage to make sure that it doesn't become an onerous burden on the owning teams. But it can be a significant empowerment in order to reduce organisational complexity, build empathy for how systems are being built, spread knowledge and expertise and documentation, and allow everyone to contribute rather than just a small subset being gatekeepers.

Hannah:
I think you have to be quite careful... We've obviously talked about DevEx being very broad and holistic and incorporating a lot of aspects. You do also then have to be careful that it's not trying to scoop up everyone and fix everything. There should be accountability held on a strong product management skill set and function as well. It can't fix everything. It's part of that wider company or project, and the checks and balances and accountability are a good thing as well.

Hannah:
I know there's certainly been, I guess... Certainly within consulting, I think... I keep going back to the consulting. It's my frame of reference. There can be a tendency towards I've seen a problem, I'm going to fix it. There has to be a balance between I've seen a problem, I'm going to fix it and I've seen a problem, I'm going to call it out and hold the right people accountable because that's their responsibility, and everyone is working towards the same end goal but doing their own functions and their own, their own things within that. I think it's a really hard balance to navigate, but there does have to be boundaries to responsibilities, as broad and holistic as DevEx may be.

Charles:
Yep. Crystal?

Crystal:
So this kind of comes back to something earlier that's kind of happening a lot in organisations around, when we talk about product operations. And there are other kind of sister functions that exist in organisations, and I think pairing with them is good. We started creating this type of thing in our organisation. And so, Charles, when you talk about like we can focus on some of the things that they feel more comfortable, they're within our control as engineers, and some of it is harder. It's harder to debug because you're debugging organisations then and processes and cross-boundary function kind of breakdowns and stuff like that. But that's why I think things like when you think about product ops and DevEx, they could be happy partners in terms of doing that kind of organisational debugging. And so that's a group that we've started to pair with very closely.

Crystal:
The other interesting pairing that we've had is Snyk being a SaaS/PaaS company, we also have DevRel, so we're trying to figure out how does DevRel function and how do we bring DevRel into DevEx? Because DevEx is kind of like DevRel function in a lot of ways, and it's so like how do we marry these kind of things that could help us under one team? But also, it's not like that one team's going to fix it all because it's a small team. And if DevEx becomes too big, you're probably doing it wrong. And it's more about how do we pair with other parts of the organisation to debug, that kind of systems thinking approach across the organisation. We haven't got there yet, I'll be honest. But it's an area that I'm keen to look at, because it's something that I've been asked about before, is like what if the problem is actually with, like you say, gathering the requirements? What if it's not very clear what we're building or why?

Charles:
Right. Yes.

Crystal:
Then we're building the wrong thing. That's very wasteful, right? And so going back to lean thinking and trying to imbue that back into the organisation. Yeah. So I don't have answers. I'm interested though, and maybe if you ask me again in a year, I'll have much better answers on how we've done this.

What are the pros and cons of having a dedicated function for developer experience versus embedding DevEx people in stream-aligned dev teams?

Charles:
There are a couple of interesting variations on the question. There's one here from Tudor, and I've had a couple of others send to me privately as well, which I'll try and merge these together. So it's basically down to what are the pros and cons of having a dedicated function for developer experience versus embedding DevEx people in stream-aligned dev teams? That's probably the easiest way to summarise.

Crystal:
I could start. So I'm really interested in stuff like this. I take things like team topologies really seriously. Because it's like, how you set up your organisation is fundamental, the operating model. And so there probably are pros and cons, I think. From my own experience, I've tried different models for different things. One thing I would say is it's not one-size-fits-all. What I have seen work well is when you have more kind of core teams like this, like there's sort of six, seven engineers around this with a product manager. That doesn't mean that they do all of it. That's where communities of practice and guilds really help organisations function at large.

Crystal:
We have that here at Snyk. So while I might be running an infra function, there are a lot of infra engineers spread across the organisation and people from SRE backgrounds. And it's like so by way of introducing something like a guild, you can still give them kind of a stronger voice for influencing the direction of where those practices or that platform or whatever it may be, where that goes, and maybe even joining on kind of a fast track. Okay, this is going to be a bit of a bumpy ride, but you're kind of our design partner because you're on the other side but you also understand our context because you've done this type of role before. So I think that that can be quite a good bridge. But yeah. But maybe the other two have more to say on this.

Suhail:
I can go next. I don't think the two are in contest, and I think you can have a developer experience function which is a bit more holistic and I guess what we call production engineers, folks who sit within their individual teams and help sort of feed back into the general developer experience function, but also are optimised for their local group. And I think that that sort of last bit is key. Because if you only have, I guess what we call the production engineer, folks who are hyper optimized within their particular group, if you look at Team Topologies and things like that, they're considered head count of the owning team. And so their priority is first their owning team. So for example, if you've got a large crosscutting change or something that could benefit many, many teams, the typical response is to deprioritize that in order to get the quick and dirty variant going to make your team quicker.

Suhail:
So there's no one really looking after the holistic picture. And where a developer experience team can excel is, their remit is to look at the holistic picture. But the two don't need to be in contest. One question that's come up which is quite similar is this developer experience team, how do you build that empathy? And sitting and embedding with the teams and understanding how they work is a core function of being an efficient developer experience team, understanding how engineers are building and writing software. You might end up in a situation where you're only impacting one particular team, and that is entirely okay. But you might find that because you have a broader remit, you empower a larger group of engineers rather than just your localised team.

Charles:
Hannah, do you want to come in on this?

Hannah:
I completely agree. I think sometimes we see the team direction as linear, but actually, particularly when you're thinking about your life service team, your production team, and then your DevEx team, you might traditionally place either end of the end-to-end scale. It's not. It's circular, and actually having them close makes all of the rest of the circle flow better.

How do you see the developer experience function evolving over time?

Charles:
So how do you see the developer experience function evolving over time?

Hannah:
I think we've had a lot of chat on this in the chat about well what does it look like in terms of integration into teams versus the kind of protection of their own time to make sure that they can do that and not get sucked into the day to day. I think it has to come down to the environment. Where does it work? What is the company set up like to succeed? Do you have full capability teams and that's the norm and they own sections of product? Do you have product teams who own very succinct pieces and need very succinct skills? You have to be able to flex. And I think the future of it is that we're better at articulating the importance of it so that it becomes a non-negotiable, but we're able to flex how we implement it based on the needs of the company, the project, the culture.

Suhail:
Yeah. I agree. I think the whole movement, the whole developer experience movement is experiencing a bit of an explosion, and there's a lot of things that are coming into the world of developer experience. So something I'm particularly keen on is the amount of things we're able to offload from the security perspective onto the general day-to-day work. Me and Crystal didn't have a chat about this at all, but things like Snyk, we're able to embed that as part of the PR process, for example, and sort spread the whole security awareness about having vulnerable dependencies and introducing dependencies which don't meet licence requirements. Things that will typically be like a burden to someone who'd do an audit once a year, like oh, here's 10 violations of GPL that you're doing by accident. Those sorts of tools are becoming more and more ubiquitous, and a lot of companies are now really talking about the developer experience.

Suhail:
It's like a fun and sexy area to be in, right? And a lot of teams are talking about their lived experiences. We're having conversations like this. I think it's a good testament, and we're able to share some of these learnings and some of our experiences, things that have worked well, things that haven't worked well in our various industries. Even in areas which have been traditionally very, very conservative, like I speak to a lot of investment banks and things like that, the engineers that are joining those companies now, they are expecting that level of developer experience tooling to be there.

Suhail:
An engineer that's joining Goldman Sachs is no longer going to tolerate Java 5.0 being their standard development environment. So these things are mattering. A lot of engineers are very in tune with what the industry is saying, and they don't want to be lagging behind. They want to be able to be alongside the big development forces like the Googles and the Facebooks and things like that. And the fact that we're having these conversations, engineers are very, very keen to stay up to date, more so now than ever, and developer experience is a very, very large part of that. And yeah, we're able to share our experiences.

Charles:
Crystal, do you want to come in?

Crystal:
Yeah, just to add onto that. So when I think back to when I started, right, I don't think it's rose-tinted glasses to say those were easier days, those were easier times. It's like FTPing something to a server and hoping that stuff's not going to go wrong. And if it does, you'll just FTP over the top of it, and there was no source control. Or I definitely didn't use it. It probably existed by that point, but that wasn't very common. And so it's just so hard to stay on top of things nowadays. Like I just ended up doing something in the office here last week around a whole bunch of people on an accelerated kind of boot camp course and like what is it like to join the industry? And for them it's tough. They are expected to know all this stuff, and it's way more than I was expected to know coming in as a computer science graduate.

Crystal:
And so I think it is about paved roads. It is what Suhail was saying there. Engineers just have a higher standard of what they expect from their environment and day-to-day work. They're not going to put up with shit like they used to. It's like they have 1,000 options to go and work somewhere else. So it's kind of like from a brand and an employee attraction point of view, we need to invest in this. And it's something that can set us apart in a big way from our competitors as well. Where it's going? I think we're just going to see more of it, is the truth. I don't honestly know what it will look like in the future. Like I said, from my own perspective, we have one team doing this right now at Snyk. I think eventually, it becomes multiple teams, is the truth, with kind of more specialisation. But also carrying on partnering with other parts of the organisation and other functional areas to try and look at how we do this kind of more holistically.

Charles:
I think that's actually a really, really nice point to wrap this up. So huge, huge thanks to my three panellists, Hannah, Crystal, and Suhail. Thank you very, very much indeed for taking the time to join us this afternoon. And for everybody who's been... We've had some great conversations going on in the chat kind of concurrently, which has been terrific. So thank you very much for this. As we said at the top, the session is recorded, so there will be a video published very, very soon. So you can look out for that. If there's anything you want to miss, sorry, anything you missed that you wanted to go back and look at. Then that will obviously be in the audio and you can share that around. Carla, are there any other last minute notices, things to say before we close?

Carla:
No, everybody who registered is going to receive an email with the video and some extra content that you can check out. And, yes, if you'd like more of it, I put a reminder on the chat, you can subscribe to our newsletter or come to our conference in person next year, even better. I just want to thank you as well, Charles, for facilitating this panel, and our panellists as well. I think it's been a great conversation with a lot of insights. We'll try to answer some of the questions that came through registration via email. There were a lot of them. We'll do our best. But, yes, thank you everybody for joining us today. It's been a blast.

Charles:
Thank you very much.

Related Content from CS

Blogs
WTF is Developer Experience and Why Does it Matter?
How Monzo’s Opinionated Platform and Tools Support their Developer Experience
How Developer Experience Portal “Backstage” Solved Spotify’s Complexity Problem
#A11y: Accessibility Is Part of the Developer Experience
How to Design an Internal Developer Platform

Podcasts
GitHub Director of Engineering, Liz Saling, on Pausing Feature Development to Focus on Tech Debt and DevEx
Sarah Wells on The Role of a Tech Director, Internal Tech Conferences, and Developer Enablement at the FT